GovLoop recently hosted an interactive online chat sponsored by the GovLoop Technology Council- How to Leverage Big Data to Improve Services. It was a great discussion moderated by Andy Krzmarzick of GovLoop and led by Bob Gourley, editor of CTOvision.com and founder of Crucial Point LLC, focused on all things Big Data. How to define it and leverage it at your agency, the future of Big Data, and its possible impact on public service and humanity as a whole.
The audience submitted many thoughtful questions and I’ve summarized some of the highlights below.
The Importance of Definitions:
Andy started off asking a simple question that proved to not be so simple: what is Big Data? According to Bob Gourley, Big Data has been discussed in tech circles for the last few years, but has recently exploded to become the new hot topic. Originating with a Google memo back in 2003, Big Data is loosely defined as datasets so large and complex, normal management systems cannot work with them. It is a new approach for companies and government agencies to do things based on a constant steam of real-time information. Why is it important to definite Big Data? Like many other hot topics, Big Data runs the risk of becoming a buzz word. If that happens, its ability to make a real difference on the way we use and analyze data decreases.
Early Adopters:
The conversation then switched gears to discuss early adopters in Big Data. While it is still in the early stages, three agencies were mentioned as good prototypes- GSA, the State Department, and NSA. Specifically, Bob Gourley highlighted GSA’s IT Department, which is delivering on mobile solutions and providing better, more cost-effective services to citizens via its search program. The State Department, Bureau of Consular Affairs, uses a suite of software applications to gather information on applicants for immigrant visas. To use all of the data it collects effectively, it stores it in the Consular Consolidated Database (CCD), a powerful, yet economical data platform. Although the specifics are largely unknown, NSA is also a good example of an agency maximizing its large source of data to improve services and operations.
Framework:
Much of the discussion was focused on Hadoop, a framework of tools that permit distributed analysis over large quantities of data, and other systems that can help support and analyze all of the data. In addition to Hadoop, which is potentially the most well-known framework, there are others, such as Splunk and larger software companies that provide similar capabilities. As Bob Gourley noted, it is also possible for IT Departments and users to build their own system, but given the large quantities of data streaming in, it often proves very difficult, especially for larger agencies and organizations. To learn more about Hadoop and the other solutions, I recommend checking out Bob Gourley’s blog, which has great information.
Challenges:
There are many challenges users and agencies face with Big Data including storing, sharing and analyzing. One discussed was the necessity of having the correct infrastructure in place to house all the data. Another was that managing and analyzing the data is incredibly complex and hands-on. As Bob Gourley noted, loading and configuring a system like Hadoop is difficult for any IT department. Writing an application for the dataset is even more time consuming and difficult. Many users struggle to manage and analyze the data, especially if they are not trained in coding or programming, so agencies must have the right personel to effectively manage the data. Finally, once the system is in place how do you keep up with the constant stream of data to make a real impact? There didn’t seem to be a “right” answer on this, except for narrowing your focus to target specific problems.
The Future of Big Data:
What is the future of Big Data? As discussed, Big Data is still in the early stages and where it’s going remains largely unknown. However, Bob Gourley ended by saying that analyzing and making better sense of all the data pouring into agencies could solve many of humanity’s greatest problems. Big Data allows agencies to filter certain information and put it into practice more efficiently and cost-effectively than ever before.
Miss the chat or want to share it with a colleague? Check out the archived event and these other good resources.
3 Challenges for Big Data and Government Agencies
This post is brought to you by the GovLoop Technology Solutions Council. The mission of this council is to provide you with information and resources to help improve government. Visit the GovLoop Technology Solutions Council to learn more.
 |
 |
 |
 |
 |
 |
 |
 |
