Agencies Should Focus on Metrics Not Measurement to Understand Cyber Impact

CISA recently released new guidance for implementing zero-trust architecture governmentwide in a push to make it easier for agencies to continue making progress in their cyber maturity. One of the keys to success will be for agencies to refine their use of metrics to better understand how to track and steer progress.

Cybersecurity deals with binary data — ones and zeros — that is inherently quantitative, and organizations usually collect a lot of data — often far more than they can effectively analyze. But in cybersecurity there is a difference between a cyber measure of that data — a raw number — and a metric which is a measure with context.

This is an important distinction because it is the difference between focusing on what happened and addressing the so what dimension that makes cybersecurity adaptive and more effective. Government needs to do more than just collect raw measurements to understand the impact of any initiative, including in cybersecurity. Metrics put measurements of cyber activity into context that the government can use to build their cyber resiliency.

Over the last 15 years, I’ve seen federal agencies try multiple options to measure their cybersecurity and cyber resilience. A decade ago, when the federal government sought to measure how much it was spending on cybersecurity, the first challenge was not in collecting data, it was deciding what was cybersecurity activity as opposed to something else — such as IT — and in ensuring that agencies measured in common terms.

Similarly, the whole of government Comprehensive National Cybersecurity Initiative I helped to oversee generated lengthy quarterly implementation reports that largely focused on measures such as numbers of security devices or threats detected — measures of what we were doing rather than metrics of effectiveness or impact.

At the same time there was a broad consensus that the federal government needed to do a better job at cybersecurity and significant resources were allocated. But our success at implementing better security was hampered because we had difficulty in measuring relatively effectiveness — and perhaps even more importantly, in understanding empirically what wasn’t working so we could determine how to fix it. Focusing on metrics instead of measures of activity arguably could have helped us identify and solve our biggest cyber problems.

If metrics are effectively designed, they help incentivize pursuing a strategic approach that focuses on the big picture rather than trying to solve every problem in a silo. This becomes especially important given the increasing prominence of cybersecurity and a new national cyber strategy that focuses on improving cyber resilience and on the roles and security responsibilities of producers and consumers of digital products and of critical infrastructure owners and operators.  

When developing metrics there are a few things agencies should think about, including:

• Intent should shape metrics. That means deciding what you want to find out and the best way to measure that, whether it be establishing a baseline of performance, demonstrating compliance with specific legal or regulatory requirements, or doing gap analysis to drive targeted cyber improvement.
• Balance simplicity and complexity. Neither extreme should be the norm. While some metrics will be complex by nature, complexity comes with a literal cost (collecting and analyzing data isn’t free, even when done in-house), and having too much data can make it difficult to “see the forest for the trees” and determine what is most critical.
• Get the most out of metrics by utilizing systematic/institutionalized partnerships. Every part of an agency generates data. The best cyber leaders leverage data collected by other departments and mission areas for their own purposes. “Not invented here” should not apply in creating cyber measures — especially when we are trying to measure cyber effectiveness across government or at the national/societal level.
• Translate from cyber speak or information-bounded metrics for senior management. Most federal leaders are not engineers or computer scientists, but it is still crucial for them to understand how cyber policies and programs are impacting agencies and their security. That means communicating metrics to them in a way they can readily understand without doing so in a fashion that appears simplistic or condescending.

While not a complete list of factors to take into consideration, it’s just enough to make agencies think about their goals and what their metrics should accomplish in terms of cybersecurity.

Establishing what types of information agencies want to gather from their data is just the first step in the development process. The next step includes preparing to gather data, which may not be as simple as it seems. With various sources and technologies readily available, leaders must choose what will best suit their mission before undertaking the collection process.

Jim Richberg’s role as Fortinet’s Field Chief Information Security Officer for the Public Sector and Vice President of Information Security leverages his 35 years’ experience leading and driving innovation in cybersecurity, threat intelligence, and cyber strategy.

Before joining Fortinet in 2019, he spent 20 years at the CIA before joining the Office of the Director of National Intelligence, where he served as the National Intelligence Manager for Cyber, the senior Federal Executive focused on cyber intelligence within the US Intelligence Community. He led the creation and implementation of cyber strategy for the 17 departments and agencies of the IC, set integrated priorities on cyber threats and served as Senior Advisor on cyber issues. He also helped create and implement the whole-of-government Comprehensive National Cybersecurity Initiative (CNCI) under Presidents Bush and Obama.

Image by VIN JD from Pixabay