FEB 15, 2017 9:24 AM PST

Coders Aim to Archive Climate Data

On Saturday coders around the country joined a hackathon organized by DataRefuge and the Environmental Data and Governance Initiative, racing to archive NASA’s climate science data before the country wakes up one day and it has miraculously disappeared. This is a legitimate fear, as the Trump administration has begun its mission of nipping and tucking certain information from public access.

Coders gather to archive climate data. Photo: Wired

The 200 coders met in Doe Library on the UC Berkeley Campus, but there were similar hackathon communities with the same intended mission in over twenty cities. The groups developed an efficient system in which hackers were split into two roles: the taggers and the baggers. The taggers were responsible for finding and marking the specific sites and data sets that needed to be archived, while the baggers were in charge of writing the code to download all the data into the Internet Archive, a digital library. “The process involves developing web-crawler scripts to trawl the internet, finding federal data and patching it together into coherent data sets,” writes Wired. This task is more difficult than one might imagine because there is essentially no consistency in the way government data has been presented on public sites in the last thirty years.

Nevertheless, when the hackathon ended the coders had successfully downloaded 8,404 NASA and DOE webpages onto the Internet Archive— essentially all of NASA's climate data. They also developed “backdoors” to download 25 gigabytes from 101 public datasets, and were expecting even more to come in as scripts on some of the larger datasets finished running, reports Wired.

But that’s not all the hackers accomplished. Figuring that this disappearing information will continue to be an ongoing crisis, the programmers are developing software that will help track the changes in websites, so that we will be aware of what we are losing and when. Engineers call this version control. For instance, the Global Data Center's reports and one of NASA's atmospheric carbon dioxide (CO2) data sets has already been removed from the web.

"Climate change data is just the tip of the iceberg," Eric Kansa, an anthropologist who manages archaeological data archiving for the nonprofit group Open Context, told Wired. "There are a huge number of other data sets being threatened [that are rich] with cultural, historical, sociological information."

Sources: Wired, Live Science




 

About the Author
  • Kathryn is a curious world-traveller interested in the intersection between nature, culture, history, and people. She has worked for environmental education non-profits and is a Spanish/English interpreter.
You May Also Like
AUG 15, 2021
Genetics & Genomics
Stunning Connection Between Bear DNA & Human Language Groups Is Revealed
AUG 15, 2021
Stunning Connection Between Bear DNA & Human Language Groups Is Revealed
Incredible findings from the wilds of coastal British Columbia have shown how closely linked animals, humans, and the en ...
AUG 30, 2021
Chemistry & Physics
Arctic Could Be "Practically Ice Free" Before 2050
AUG 30, 2021
Arctic Could Be "Practically Ice Free" Before 2050
The Intergovernmental Panel on Climate Change’s (IPCC’s) recent physical climate science report details the ...
SEP 05, 2021
Genetics & Genomics
New Comb Jelly Species Revealed by Genetic Study
SEP 05, 2021
New Comb Jelly Species Revealed by Genetic Study
Genetic research has now revealed more about comb jellies, also known as ctenophores, which are delicate and mysterious ...
SEP 09, 2021
Chemistry & Physics
"Shape-Shifting" Birds: Climate Change's Newest Outcome
SEP 09, 2021
"Shape-Shifting" Birds: Climate Change's Newest Outcome
A new study released last Tuesday highlights a surprising response to climate change: physical changes in animal mo ...
SEP 27, 2021
Plants & Animals
Nearly 90% Of Amazonian Plants And Animals Have Been Affected By Wildfires
SEP 27, 2021
Nearly 90% Of Amazonian Plants And Animals Have Been Affected By Wildfires
Wildfires and increasing levels of deforestation in the Amazon rainforest are not new challenges; in fact, Brazil has lo ...
OCT 06, 2021
Chemistry & Physics
Chemistry and Magic: Identifying the Oldest Records of Merlin the Magician
OCT 06, 2021
Chemistry and Magic: Identifying the Oldest Records of Merlin the Magician
Historians may have found the earliest manuscripts that tell the story of Merlin the Magician. Merlin is a character fro ...
Loading Comments...