FEB 15, 2017 9:24 AM PST

Coders Aim to Archive Climate Data

On Saturday coders around the country joined a hackathon organized by DataRefuge and the Environmental Data and Governance Initiative, racing to archive NASA’s climate science data before the country wakes up one day and it has miraculously disappeared. This is a legitimate fear, as the Trump administration has begun its mission of nipping and tucking certain information from public access.

Coders gather to archive climate data. Photo: Wired

The 200 coders met in Doe Library on the UC Berkeley Campus, but there were similar hackathon communities with the same intended mission in over twenty cities. The groups developed an efficient system in which hackers were split into two roles: the taggers and the baggers. The taggers were responsible for finding and marking the specific sites and data sets that needed to be archived, while the baggers were in charge of writing the code to download all the data into the Internet Archive, a digital library. “The process involves developing web-crawler scripts to trawl the internet, finding federal data and patching it together into coherent data sets,” writes Wired. This task is more difficult than one might imagine because there is essentially no consistency in the way government data has been presented on public sites in the last thirty years.

Nevertheless, when the hackathon ended the coders had successfully downloaded 8,404 NASA and DOE webpages onto the Internet Archive— essentially all of NASA's climate data. They also developed “backdoors” to download 25 gigabytes from 101 public datasets, and were expecting even more to come in as scripts on some of the larger datasets finished running, reports Wired.

But that’s not all the hackers accomplished. Figuring that this disappearing information will continue to be an ongoing crisis, the programmers are developing software that will help track the changes in websites, so that we will be aware of what we are losing and when. Engineers call this version control. For instance, the Global Data Center's reports and one of NASA's atmospheric carbon dioxide (CO2) data sets has already been removed from the web.

"Climate change data is just the tip of the iceberg," Eric Kansa, an anthropologist who manages archaeological data archiving for the nonprofit group Open Context, told Wired. "There are a huge number of other data sets being threatened [that are rich] with cultural, historical, sociological information."

Sources: Wired, Live Science




 

About the Author
  • Kathryn is a curious world-traveller interested in the intersection between nature, culture, history, and people. She has worked for environmental education non-profits and is a Spanish/English interpreter.
You May Also Like
SEP 15, 2020
Microbiology
If They Must, Methane-Eating Microbes Will Consume Ammonia
SEP 15, 2020
If They Must, Methane-Eating Microbes Will Consume Ammonia
There are many different kinds of microbes, and some can use unusual substances to survive. Methanotrophs, for example, ...
OCT 07, 2020
Earth & The Environment
The conversion of the Amazon rainforest into savanna may be on the horizon
OCT 07, 2020
The conversion of the Amazon rainforest into savanna may be on the horizon
More dim news from the Amazon: more of the rainforest is at risk of becoming a savanna than previously believed. The fin ...
OCT 15, 2020
Plants & Animals
Meet the World's First Cloned Przewalski's Horse
OCT 15, 2020
Meet the World's First Cloned Przewalski's Horse
Say “hello!” to Kurt, a two-month-old Przewalski’s horse that has made scientific history as the world ...
OCT 15, 2020
Health & Medicine
An Estimated 38 Million People were Exposed to Polluted Wildfire Smoke
OCT 15, 2020
An Estimated 38 Million People were Exposed to Polluted Wildfire Smoke
As wildfires continue to burn record-breaking acreages along the west coast of the United States and inland regions, hos ...
OCT 18, 2020
Earth & The Environment
Pandemic-enforced lockdowns cut premature deaths from air pollution
OCT 18, 2020
Pandemic-enforced lockdowns cut premature deaths from air pollution
A new study published in The Lancet Planetary Health reports that pandemic-enforced lockdowns in China and Europe h ...
NOV 17, 2020
Plants & Animals
Taking a Virtual Walk in a Forest Altered by Climate Change
NOV 17, 2020
Taking a Virtual Walk in a Forest Altered by Climate Change
For some people, it's easier to grasp the magnitude of a problem once it can be visualized.
Loading Comments...