The President’s Management Agenda provided agencies with a clear goal: Leverage data as a strategic asset. In theory, they are in great positions to do so. They have growing volumes of data available to use and develop actionable intelligence to drive decision-making.
Most federal agencies, however, store much of this data in legacy systems, some of which are decades old. These systems create information silos that make it difficult for agencies to share and analyze their wealth of data and create a “single source of data truth.” Without a single source of data truth, misinformation can spread, which can hurt public trust and hinder agencies’ abilities to perform critical missions.
That’s not to say agencies don’t try. Inspired and resourceful data experts will find tools and processes to work around those barriers and get their work done. But that’s far from an ideal scenario.
“If you spend all of your time developing the tools to answer questions, the motivation to keep asking questions becomes lesser and lesser,” said Nicholas Speece, Chief Federal Technologist at Snowflake.
The data environment itself has also grown more complex. About 10 years ago, the emergence of the Internet of Things (IoT) and new datasets generated by sensors led to the development of new platforms such as Hadoop and Google Big Query. These platforms could analyze a lot of data simultaneously but couldn’t scale. Additionally, they were tough to develop applications for because they used specific languages and required developers to have targeted skills. Security is also a concern for agencies as they look to secure their data without it becoming an obstacle to data-sharing.
The Solution: A Purpose-Built Cloud Data Platform
Whether it’s different departments within an agency, different agencies or even different state and local governments, data is currently not always easily shareable or quickly accessible to those who need it most.
The key to addressing this challenge is to move data to a cloud data platform, where all data from across the organization can be accessed, shared, analyzed and protected in one location – a single version of truth.
For that to happen, agencies need to take a different approach to data management. In a traditional legacy environment, storage, compute power and data pipelines are interwoven, which creates silos, whether on premises or in the cloud.
“Those three things should be completely independent and decoupled, yet still tightly integrated,” Speece said. “They’re the three biggest components of getting data into the cloud. If I’m going to take data from seven silos in my organization, I don’t want to just create seven new silos in a cloud server.”
In a modern cloud data platform, storage, compute and services layers logically integrate but scale infinitely and independent of one another. That decoupling has important financial ramifications. Traditionally, buying more compute means getting more storage because they’re connected. A cloud data platform lets agencies efficiently build an environment that can scale a single component without affecting others.
A cloud data platform serves the middle ground between data lakes — where raw data is stored in bulk — and data warehouses that put data through a transformation process, normalizing it so that it’s easily and readily consumable. In a data warehouse, agencies can combine semi-structured data such as Extensible Markup Language (XML) and JavaScript Object Notation (JSON) with structured data in one place to enable ad hoc queries separating storage from compute.
Cloud-ready data platforms can scale, access more sources and incorporate better analytics, which leads to better and quicker decision-making. This also breaks down both intraagency and interagency silos because data stored in the cloud can move among agencies. Finally, security is improved and simplified in a cloud environment.
This article is an excerpt from GovLoop’s recent report, “The Need for a Single Source of Data Truth.” Download the full report here.
