This blog post is an excerpt from GovLoop’s recent Industry Perspective, Preparing for the Data Deluge in the Public Sector. Download the full Perspective, here.
Several issues affect data management today: the sheer amount of data that currently exists and the rate at which new data is being created; a record demand for quality data to make informed decisions; and the difficulty end users have accessing and sorting through the data they need.
Like puzzle pieces, these issues taken together reveal the whole picture, but to get there, we must first examine what’s shaping each.
The Data Landscape: Growing, Sprawling, and Hard to Manage
One reason why data is doubling every three years is that agencies are simply creating much more of it.
“Everything from end users creating it to sensors to airborne platforms — you name it, it’s more data than ever,” said Warnk, Director of Data Protection and Management at ViON. “There are so many vectors for data these days. Twenty years ago, not so many systems were creating data. Now, almost everything in your environment is creating information, including the concept of the Internet of Things, where there’s a lot of machine-written data.”
Warnk is right. The public sector is seeing the most explosive data growth in human history, and it’s possible that this trend will continue to accelerate exponentially. The amount of data that today’s IT departments must store, manage and protect is massive. Over the course of the last 10 years the market has seen a dynamic shift where the growth of unstructured data – including email, images, video, social media, documents and more – is outpacing the growth of storage systems.
In addition to the massive amounts of data being created, other factors are at play when it comes to where data lives, how it’s accessed, and how it’s retained.
For example, the rise of mobile devices and employees using multiple, personal devices means that data is scattered over a wide variety of devices that can be difficult to index and access. Today in the public sector, employees no longer use only agency-issued smart phones, tablets and laptops. For the past few years, they’ve pushed to bring their own devices. Now many use their personal devices to conduct official business, including creating, storing and sharing government data. The result is that data could reside anywhere – even in places agencies don’t know about – raising additional questions about security, compliance with retention policies and storage.
Additionally, compliance policies and mandates play a part in data management because much more of the data created today must be retained long-term and in an easily accessible format. One example? To account for digitization of records, President Barack Obama signed in 2011 a memorandum on government records management that requires federal agencies to manage all permanent electronic re- cords in a digital format by Dec. 31, 2019, and all permanent and temporary email records in an accessible electronic format by Dec. 31, 2016. Among other things, the law “clarifies the responsibilities of federal government officials when using non-government email systems.”
This perfect storm of data creation and sprawl has to be dealt with in a world where the fact is that data is increasing between 37 percent and 77 percent a year but IT budgets are increasing between 2 percent and 4 percent a year.
