The Astronomical Growth of Data

This post is an excerpt from GovLoop and Quantum’s recent industry perspective, “7 Ways to Unlock the “Big” in Your Big Data.”

Before diving into how to manage big data, it’s important to understand why so much data is being amassed in the first place — because the drivers of growth also determine how it can be tackled.

The exponential increase in the amount of data being created has many organizations struggling to keep up. However, Claire Giordano, Senior Director of Emerging Storage Markets at Quantum, sees a difference in big data and big data analytics: “Beyond the analytical challenges, I’m talking about solving the complex data management challenges that come with big scale.”

Several factors are contributing to the increase in big data:

Raw data growth. The evolving sophistication of technology is simply creating more data than ever before. Cameras and sensors can be placed on almost anything today, from drones to buildings to satellites, creating a wealth of data to power mission-critical decision making.

Advanced analytics capabilities. Once that initial data is captured, advanced analytics technology lets scientists and government users do more with it. However, the software analysis creates new, unique datasets that also require processing and storage.

Increased retention. Because technology advancements have created new ways to repurpose data, more data is being preserved for potential future use even when it is not considered valuable today. “Many government agencies have very long retention periods – scientific research, intelligence and military use cases for example. In fact, when you ask life sciences researchers what the retention timeline is for their data, they will say that it’s indefinite,” Giordano said.

Additionally, legislative mandates and compliance regulations require more agencies to maintain detailed archives of their past operations and data. “Organizations are simply retaining data longer,” Giordano said.

A Solution to Big Data Growth

Multi-tier storage — a solution where data can be stored on different types or “tiers” of storage based on the needs of the application and the workflow — is an approach being used more and more in big data environments. The flexibility to automate the movement of data between types of storage such as flash, high-performance disk, object storage, tape archives and cloud can be a key enabler to unlocking the value in big data, Giordano said.

To learn more about the benefits of multi-tier storage, check out our industry perspective, “7 Ways to Unlock the “Big” in Your Big Data.”

Photo Credit: NASA Goddard Space Flight Center