3 Data Storage Technologies: A Comparison

Public servants at all levels of government face tough questions around the future of service and operations.

The increased use of facts and evidence over hunches and anecdotes during the pandemic set leaders at the state and local levels up for success after the crisis. Challenges remain, however, in effectively using technology tools to mine available data for useful, actionable information into the future. Barriers to full data use — before stakeholders even access it — include data quality, timeliness and relevance.

To support successful programs for decision-makers, researchers, analysts and the public, data leaders typically choose data lakes or data warehouses for their program needs. These options essentially offer data storage at scale, with the goal of breaking down data silos and providing access to data that would otherwise be stuck in source systems.

For government leaders looking to find a scalable data solution, a third option exists: data platforms built specifically for the public sector. Such vertically dedicated platforms are gaining traction among some of the most innovative local, state and federal agencies in the U.S.

A closer examination of these three options is helpful in determining which is right for your organization and its strategic goals.

Data Warehouses

A data warehouse is a system to collect structured, historical data from siloed data records. Like goods in a warehouse, this is a highly organized and cataloged storage configuration.

This technology is designed to store structured data that have a pre-determined schema, which is directed by the specific use cases for the data. Data warehouses capably support data analysis as well as the use of business intelligence (BI) tools, reporting and building visualizations with aggregated data.

Given their highly structured nature, they are less suitable for easy self-service access for non-technical personnel. The advanced skills of a database analyst who understands the written schema of the data are usually needed to work with and derive value from a warehouse structure. Access and rapid time-to-outcome for pre-determined analytics are possible, so long as the right experts are on hand to navigate the complex architecture. Tools (such as data marts) and expertise are also needed to extract, load and transform (ETL) data before it is ready for reporting. To arrive at actionable insight, the data often travels from the warehouse to a mart to an operational system or reporting tool, all of which come at additional costs.

Data Lakes

Data lakes are systems in which large volumes of raw data are stored in their unstructured and natural states. Like a lake filled with water, the data lack structure and organization. The lake serves as a “dump” for all operational and transactional data.

Because data lakes store fluid, unstructured and structured data, they can store large volumes at a low cost. The volume and lack of structure can make it complicated to gather the data for a specific use. The data, for instance, has to be transformed before it can be analyzed or applied to answer a question or solve a problem. Access to a technical team — a team with enough time to contribute — is crucial to reduce the typically high time-to-outcome for insights and analysis.

Data lakes should be accessed only by highly technical users to avoid the risk of significant errors that can impact data governance, security and ETL processes. Without quality enforcement, there can often be challenges with data flow, which in turn makes data more error-prone, increases troubleshooting time and increases time-to-outcome for use.

Data Platforms

Enterprise data platforms that are purpose-built for the public sector support government decision-makers who want to lead with data. Unlike data warehouses and data lakes, data platforms streamline the processes of storing, transforming and sharing data. This creates a low time-to-outcome for government-specific analysis and broadens the audience of that data from a few skilled analysts to a range of stakeholders.

Data platforms can help agencies and departments take control of their data in a few key ways. Specifically, they:

• Facilitate self-service access and discovery of data
• Democratize data for a range of technical abilities
• Enable the infinite reuse via application programming interfaces (APIs)
• Share contextualized data internally or with the public

These functionalities help governments at all levels apply data to a range of initiatives such as increasing equity, improving service delivery and reporting on COVID-19 recovery.

Data platforms developed for government inherently solve the complexities of data-sharing and strict privacy requirements. At the same time, such platforms maintain a simplistic user interface (UI) that makes a data query as easy as a Google search, reducing the barriers to working with data and putting it into meaningful action faster.

Determining Your Needs

These fundamental questions can help guide which approach makes the most sense for your data program:

• How important is data to your organization?
• How confident are you in your ability to make data-driven decisions?
• How many employees manage and analyze data on your team?
• How do those employees share insights across the organization?
• Would your decisions have more impact if you had access to real-time data and collaborative insights?

Data is certainly needed to manage crises. But beyond that, it can be a catalyst to increasing functionality within your organization for stronger internal performance and better community outcomes. Enhanced access to data improves every decision every day, making this an area of strategic priority for government leaders seeking innovation and long-term stability.

Meredith Trimble is a former municipal official and Town Council Acting Chair, who focused on strategic planning, annual budgeting and bonded infrastructure projects. Her government experience also includes posts in both federal and state-level executive branch agencies: Associate Editor of the Federal Election Commission’s FEC Record; and Director of Education for the CT Office of State Ethics. In her current role as a Content Manager, Editorial with Tyler Technologies, Inc., she writes content to help empower those who serve the public. Her current focus is to help facilitate data-enabled organizations and create connections between governments and those they serve.

This post first appeared, in slightly different form, on April 28, 2021.