Discovering Data in Today’s Government

This blog is an excerpt from GovLoop’s recent industry perspective, Discovering Data in Today’s Government. Download the full perspective here.

Government organizations face growing volumes of data and increasing pressure to provide information more transparently. At the same time, they must maintain the privacy of citizens’ data, reduce costs and provide user-friendly digital services and interfaces – letting users access data and services at the touch of a finger.

Much of the data government stores that would help meet these needs, however, is unknown or nearly impossible to find because it is in disparate areas or siloes – in archives, file shares, outdated and legacy systems, employees’ computers and other hard-to-reach places. This unknown data represents a vast source of costs, risks and opportunities. It also causes public sector employees and citizens who need to access this data quickly a variety of problems.

The solution may lie in technology that has traditionally been used for legal eDiscovery. This technology is capable of directly indexing virtually all forms of unstructured information, in any language, and processing complex proprietary formats.

To understand how eDiscovery technology can help government agencies search, investigate and better manage structured and unstructured data, GovLoop partnered with Nuix, a leader in solving data challenges—one of which is eDiscovery—for this industry perspective.

In the following pages, we will lay out how leaders at Nuix recommended the government can address working with big data volumes and complex file formats, find fast answers to data challenges and discuss how to consolidate data into searchable and manageable intelligence – no matter the size or complexity of the data set.

The Data Challenges Government Faces Today

As technology continues to evolve and advance at an unprecedented pace, new and unforeseen data management and indexing challenges are emerging. Personal computers have been supplanted by smartphones and tablets, and storage capacity in these types of devices is routinely measured in the tens or hundreds of gigabytes and, in some cases, even terabytes. These are volumes that just a few years ago were only found in servers.

While government is reaching new heights with this data and the information and insights it provides, many agencies are still struggling with a number of issues, particularly in data structure, data integrity and processing capabilities. Making sense of all the data that government must collect and share is hard enough for any one individual, let alone an entire organization. And specifically for government agencies, today’s explosion in data growth is fueling greater demands for solutions that can help them analyze data and ultimately make better decisions.

But achieving that goal is no small task, due to a variety of reasons, including insufficient manpower, the sheer volume of data, a variety of mandates that agencies must now comply with and insufficient time.

“For the inquiries that are coming in, organizations need to be able to respond in a timely fashion,” said Stephen Stewart, Chief Technology Officer at Nuix. “And accordingly, the expectations of response from citizens have also increased. But the amount of time allowed for them does not also increase relative to the amount of data that an organization has under its control.” Many agencies are facing records management challenges that affect their ability to respond effectively to Freedom of Information Act (FOIA) requests. Those challenges are resulting in severe backlogs that are frustrating for the responding agencies—and also to the people making the requests.

The 2011 Managing Government Records Memorandum and the 2012 NARA Records Management Directive required agencies to begin managing their email records in an accessible format by the end of 2016. By 2019, they must retain all electronic records in a digitally accessible format for preservation by the National Archives and Records Administration. And more recently, the proposed OPEN Government Data Act would require all federal agencies to publish their information online, using nonproprietary, machine-readable data formats.

These mandates, the increasing number of FOIA requests from citizens, the ever growing data volume and shrinking budgets and skillsets are all combining to create eDiscovery, investigations and records management challenges that even spill over into the realm of cybersecurity. After all, if you don’t know what kind of information you have on your systems or you haven’t prioritized it properly, how can you ever begin to protect it?

“The ability to adapt and deploy new systems moves nowhere near the speed that the data has been stacking up,” said Joe Babineau, Senior Solutions Engineer at Nuix USG. “Agency leaders can’t get new systems demoed to them as fast as their data’s growing.”

To learn about the solution to these challenges, download the full perspective, Discovering Data in Today’s Government.