Federal Data Finally Gets Its Time in the Sunlight

What might be the largest index of government data in the world is about to come to light. How did this radical big data reveal come about?

Back in May 2013, President Obama signed the Open Data Executive Order to make government data more accessible to citizens, companies, and entrepreneurs. The order required agencies to create indexes of the datasets they have and then share those indexes with the public in open, machine-readable formats.

Knowing that government agencies would need something more instructive, the Office of Management and Budget (OMB) issued an implementation guide in November 2013. The guide included information on Enterprise Data Inventories (EDI) that agencies must create and release publicly to help everyone “develop a clear and comprehensive understanding of what data assets” the agency possesses.

EDI = List of a government agency’s datasets. Not the data itself, which might itself be private for security or other reasons. Clear?

The problem with the EDIs was that though agencies were required to create, maintain, and regularly release the lists to the public on their websites, few agencies complied and even then only released partial lists that didn’t include any details about private datasets.

This meant that the public had no way of knowing what data the government had and so wouldn’t be able to advocate for whether certain data should actually be kept secret (or kept at all, though that’s another matter).

The Sunlight Foundation, a nonpartisan nonprofit that advocates for open government and accessible data, saw the terrible Catch-22 in this scenario.

How would the public know what it didn’t know?

That’s nothing a little Freedom of Information Act (FOIA) request couldn’t fix, right? So, in December 2013, The Sunlight Foundation filed a FOIA request with the OMB to release all the EDIs it receives from government agencies each quarter.

Months later, the OMB responded to the Sunlight Foundation’s FOIA request. To sum up:

In its denial, the OMB recommended that the Sunlight Foundation contact each agency individually, even though the OMB was supposed to serve as a central repository of the datasets. With over two dozen agencies supposed to release EDIs on a regular basis, this seemed like a lot to ask of the public.

The OMB also evaded the problems of how most agencies were failing to post their EDIs at all and that there was no requirement for agencies to share their entire dataset inventories.

Undaunted, the Sunlight Foundation filed an appeal with the OMB and experimented by filing FOIA requests with four of the agencies that had slacked off. Props to the Department of Transportation (DOT) for being the only agency to publish their EDI.

More than a year after the first FOIA request and with the Sunlight Foundation considering legal action, the OMB responded to the appeal with a more open view on open data.

The OMB agreed to promptly release the EDIs it receives and, according to the Sunlight Foundation, acknowledged “the value of data indexes, working with agencies to develop them and, now, releasing them publicly.”

Rightly declaring this decision a big win, the Sunlight Foundation has again successfully fought for more open government data.

Adding an extra shine to this good news, around the same time a group of nonprofit and private sector organizations launched a coalition to help the federal government modernize its records management infrastructure and implement information governance practices. Or, in layman’s terms, do better with big data.

The Coalition for Public Sector Information Governance Leadership, working with the National Archives and Records Administration (NARA) and the OMB, will help agencies meet digital records requirements and also “increase awareness about the benefits of creating an information governance mindset and standards of conduct.”

The coalition is just getting started and federal agencies have until February 28, 2015 to submit their EDIs to the OMB. The OMB will have to ensure that all agencies actually submit their EDIs and then will share the EDIs with the Sunlight Foundation.

Here’s hoping that this new, refreshing cooperation for open data keeps its groove going.

Lauren Girardin is a marketing and communications consultant, writer, and trainer. Find her on Twitter at @girardinl.