Big data is driving game-changing innovations in both the private and public sector. With the emergence of the Internet of Things (IoT), the shared economy, and advances in robotics, our society is being fundamentally changed by technology and the data generated from that technology.
These developments are powering changes in the ways organizations collect, store and process data. Today, the public sector has realized that leveraging information in new ways is the key to innovation, and helps organizations improve public sector mission outcomes.
“Data needs to be treated as an organization’s most valuable asset,” said Amy O’Connor, big data evangelist, Cloudera, in a recent interview with GovLoop. Although big data has become the driving force behind modernization, big data programs often face the challenge of balancing innovation against risks. That’s why the Chief Data Officer (CDO) is now a strategic imperative for government agencies. It’s no longer a matter of if but rather when your agency will need to hire its first CDO.
The CDO as a Strategic Imperative for Government
The 2014 IDG Enterprise Big Data research study showed that organizations are witnessing rapid growth in the volume of data managed, and information is expected to increase by 76 percent within the next 12-18 months.
“We now have the technology that lets us store data and process [information] where we never could before,” said O’Connor. “And we now have the ability to learn far more about trends and objectives than we ever could in the past. This provides a tremendous opportunity for innovation – an opportunity to build new products and services that make the lives of people better.”
With this kind of expected growth in data and opportunity for innovation, an organizational need has become clear. Across sectors, organizations are looking for the CDO to balance business and technology risks to capitalize on the promise of big data.
“The CDO role has emerged because we have the ability to collect and process so much data in order to deliver new innovative products or services,” said O’Connor. “But we have to balance the risk of using data to ensure we are using that data appropriately – in ways people expect.”
As O’Connor alludes to, advanced services that users can opt into – like tracking our health or monitoring energy use of our homes – is only the tip of the iceberg for big data applications. But these quickly evolving data-driven services have led to challenges for both legislators and civil servants.
The challenges for both are very similar: create structures that protect confidentiality, but do not inhibit innovation. For government, the CDO will play an essential part in helping agencies navigate new and complex legislation to assure compliance and drive business transformations.
It’s clear that government leaders have recognized that data is fuel for innovation, and can serve as a means to rapidly provision new business- and mission-critical applications to improve the economic viability of communities with improved decision-making. The CDO will play a vital role in how to use data as the gateway to modernize government. So what does the role of the CDO within a government agency look like?
The Role of the CDO
Both the public and private sector are moving forward in the hiring of CDOs. In February, Wells Fargo hired Charles Thomas as chief data officer. And in March, the city of San Francisco hired Joy Bonaguro as the city’s CDO.
Bonaguro’s position was designed to promote transparency through data, create strategies for citywide data, and leverage analytics to make data-driven decisions. (The listing for the position can be found here.)
The San Francisco use case shows that a public sector CDO must possess skills beyond technical expertise. A CDO must have an innovative mindset and a firm understanding of how data can play a role in changing how the agency does business.
In doing so, the CDO role can help agencies take a strategic approach to leverage their information, helping organizations map their mission need to their data. “A CDO is going to enable the agency to [leverage data] to meet the world’s next set of opportunities,” said O’Connor.
In order to start this process, a CDO needs to think of ways to consolidate data that traditionally is spread out across various sources. Additionally, they will also have to find out what data is needed by the agency to make improved decisions, both now and in the future.
Above all, the CDO will have to sell big data internally to senior leadership. This means they must be able to show its ROI. “A simple way to start showing ROI is on an operational efficiency use case,” said O’Connor.
“Somewhere you are processing data in another system, and it’s just taking way too long. By simply moving that data into a Hadoop big data system, and moving the processing over, you decrease your storage cost. We usually see between 10 and 100 times lower cost from some of the traditional storage systems. You will also reduce your processing windows – that’s all about opportunity cost. Now your agency has more money to spend, since you’re not spending it on storage, and more time to actually use the data that has to be processed.”
Since the CDO is still an emerging role, some have questioned where in the organization the position should report. Current trends point to the CDO reporting to the CIO. O’Connor takes a different approach.
“I see the CDO position merging with the chief strategy officer role. It should not be under the CIO, but maybe that is an okay place to start and incubate. Ultimately, it should be right at the center of the C level table with respect to defining strategy.”
Leveraging an Enterprise Data Hub
When adopting a big data program, government needs the peace of mind to know they have a technical solution that helps them balance risk. That’s why many government agencies are looking to adopt a data management architecture called the enterprise data hub (EDH). With an EDH, data can be stored in its original fidelity, integrated with existing infrastructures and supporting the flexibility needed for various kinds of workloads – such as batch processing, interactive SQL, enterprise search and advanced analytics.
Since the EDH is powered by Hadoop, agencies can now capitalize on the ability to leverage their data in transformative ways. The Hadoop ecosystem of tools handles extremely large data processing needs at lower cost and extreme scale. Additionally, since the EDH and underlying Hadoop technology allow the data to be used in many different ways,
With big data, information is often transformed and modified to show different insights and drive improved decisions. Also, agencies still have immediate access to the original data, so they can easily go back and look at information in different ways.
But these insights can also have a dark side. Without the proper data management and governance, leaders will not know what transformations or changes were made to the data. With government collecting such authoritative data, agencies need a way to manage and track that information to ensure proper usage and veracity.
That’s why Cloudera offers Cloudera Navigator, a data management application for Hadoop. Navigator is designed to provide management capabilities for administrators, helping them to secure, govern and explore the data within an enterprise data hub.
“That’s why a tool like Navigator is so important,” said O’Connor. “Since you can transform data multiple times, you clearly have to track its lineage so that you know what happens to it. You also have to be able to audit the data to see who touched it and when.” With Navigator, a CDO is able to balance risk with innovation.
The CDO role will soon become critical to forward-looking companies who want to drive innovation. Even for organizations taking a more conservative approach to big data, the CDO offers the chance to assure compliance in a complex and changing regulatory environment. In both cases, the CDO is going to be an essential leadership role within an agency, charting data-driven strategies and maintaining the highest data standards, all while powering innovation and capitalizing on governments authoritative data.
Cloudera is revolutionizing enterprise data management with the first unified Platform for Big Data: The Enterprise Data Hub. Cloudera offers enterprises one place to store, process and analyze all their data, empowering them to extend the value of existing investments while enabling fundamental new ways to derive value from their data. Learn more here.