We live in a society bursting at the seams with data. For public sector organizations, it is important that they do not let their data go to waste. Big data analytics provides the opportunity for governments to leverage data and make evidence-based decisions. In a recent report, “Realizing the Promise of Big Data: Implementing Big Data Projects,” the IBM Center for The Business of Government and Arizona State University Professor Kevin Desouza explore the emerging world of big data and its potential for revolutionizing the public sector.
Big data is a buzzword that has increasing popularity, but is not always understood. “Big data is an evolving concept that refers to the growth of data and how it is used to optimize business processes, create customer value, and mitigate risks,” Desouza explains. “For public managers, big data represents an opportunity to infuse information and technology into the design and management of organizations, personnel, and resources.”
Desouza explains big data as a series of 7 “V’s”: volume, velocity, variety, viscosity, variability, veracity, and volatility. The most challenging for the public sector is variety. Many organizations struggle to find the right methods of integrating data from newer platforms into legacy systems.
Although extracting meaningful information from legacy systems is a challenge, there are many positive examples of big data adoption in the public sector. In the report, Desouza explores cases from the U.S. Postal Service, IRS, the state of Massachusetts, and more. He delves into a particular case study concerning New York City and how local municipalities leverage datasets to enhance management of transportation, utility, and infrastructure systems. New York City’s Business Integrity Commission (BIC) administers licenses to sanitation haulers and wholesalers. The agency regulates more than 2,000 businesses throughout the city and must ensure that waste is collected, hauled and disposed of properly.
The increased value of recyclables and yellow grease – which can be made into biodiesel fuel – caused a surge in criminal waste collection; unlicensed individuals were illegally collecting and selling these products for a profit. To stop this behavior, BIC and the Mayor’s Office of Analytics collaborated with the Department of Health and Mental Hygiene DOHMH) and Department of Environmental Protection (DEP) to leverage industry data on grease production, restaurant permits and sewer backups. By cross-referencing this data, the agencies created a yellow grease heat map to identify potential “hotspots” of unlicensed waste collection activity. BIC reported a 30 percent increase in discovered violations. Also, BIC has witnessed a 60 percent cutback in necessary manpower for grease enforcement since implementing this technology.
As the above example shows, big data analytics can revolutionize public sector management. However, it is not that simple. In a survey of Chief Information Officers (CIOs) from all levels of government, Desouza identifies 10 findings that speak to the current state of big data in the public sector. Some of the major findings indicate “public agencies are in the early days of their big data efforts” and not many CIOs “anticipate significant investments in technology.” On the other hand, CIOs believe that collaboration, leadership and working groups are necessary for additional big data programs to be adopted. Desouza notes that CIOs are “becoming champions of analytics and evidence-driven decision-making.”
The report concludes with a set of key steps and best practices to overcome the unique challenges public officials face when implementing big data projects. The implementation process is broken up into three stages. The first stage is planning. CIOs must “do their homework,” seek out peers’ expertise, and ensure that the project is in alignment with other agency efforts. Furthermore, it is important to develop process and outcome measures and assess risk and privacy concerns before starting a data project. Begin with the “lowest-hanging fruit.” Desouza observes that small opportunities are the easiest to tackle before moving on to larger tasks.
The next stage is execution. The most crucial aspect of this stage is communication. CIOs should constantly check in with all parties involved and gauge the pulse of the program. It is important to remain focused and to manage scope creep. The final stage is post-implementation. This involves conducting postmortem analysis and determining the impact of the project. CIOs can assess the lessons learned and identify the next project.
Big data is becoming more pertinent for public managers. Agencies will continue to wrestle with integrating big data analytics into their current IT systems and strive to make data-driven decisions. The IBM Center for the Business of Government’s “Realizing the Promise of Big Data” report offers helpful resources for public organizations navigating the path to implementing successful big data projects.
For more information, download IBM’s entire report here.
An item on the CBC Radio newsmagazine show “As It Happens” last night profiled a couple of researchers and avid fishermen that were able to use “big data” to corroborate that, yes indeed, muskelunge are about 5% more likely to bite when its a full moon. I’m sure fishermen heard it a different way than I did, but my first thought when the folks were interviewed and described the massive amount of archival information they employed to finally put some received fishing wisdom to empirical test was “Hey, that’s Big Data at work!”.
The segment can be heard here: http://www.cbc.ca/player/Radio/As+It+Happens/ID/2460572177/?sort=MostRecent
Of course, in the same episode, they interviewed a Harvard graduate student who operates a site devoted to examining spurious correlations.
Incidentally, I didn’t mention this to be glib, but to simply note that “big data” can show up in places where you least expect it to show up, and answer questions you thought you’d never have a means to answer.