The DOD caught with its Pants down – A Revolution in Risk Assessment needed?

The DOD caught with its Pants down – A Revolution in Risk Assessment needed?

It seems to have passed commentators by, but the US Government Accountability Office (GAO) recently uncovered a major under-reporting of risk at the Department of Defense. (see Figure 1 below).¹

The GAO’s recommendations read like a school report: the DOD should re-do its homework and try harder in the future…

However, I have a different and more revolutionary approach to risk management to propose – and it is based on real world feedback, agile style.

Figure 1: All is well, apparently…

Background

The GAO’s recent report on the usefulness of the Federal Information Technology Dashboard uncovered the fact that the DOD is:

“masking significant investment risks, has not employed its own risk management guidance, and has not delivered the transparency intended by the Federal IT Dashboard.”

The Federal IT Dashboard was introduced in 2009 and whilst it has increased the transparency of the performance of major federal IT investments, the GAO appears unsure of its usefulness. It notes that although agencies adjust the risk assessments of their investments over time, there is not enough evidence as to whether this is just because of changes to risk assessment processes, or because of actual changes in the real underlying risks.

In this latest report the GAO looked at six departments, and although the DOD whitewash of its risk assessments is the most egregious, the other departments (with the possible exception of the Department of the Interior – DOI) look suspiciously optimistic also (see Figure 1).

The GAO’s Recommendations?

The GAO has called, yet again, for more information on investment performance and more care in the rating processes used for risk assessment.

We have heard these recommendations before. In a recent presentation to senior government advisers in London I highlighted how increasing detail in planning and analysis at the FBI led to no greater insight into risk management.

In annual assessments of the FBI’s case management overhaul which in the end wasted over $600m, the GAO had recommended more and more detailed planning and risk management. They were encouraged by the mountains of paperwork that had been produced. But the program failed, despite ‘Green’ risk assessments by an ‘independent’ $100m Programme Management Office (PMO) and the annual GAO scrutiny just highlights the problem of trying to predict risk by ensuring bureaucratic conformity rather than delivering. Bureaucratic conformity, after all, is the best way to ensure ‘groupthink’ and optimism bias.

In the end, only an ‘agile revolution’ at the FBI saved the day after the prime contractor had failed to deliver again and again.

Figure 2: Brian in PS Panel Session after his talk (transcript of talk here)²

We Have Heard this all Before

This recent GAO report now repeats similar advice. The GAO have found that the DOD risk assessment rates none of its 87 current investments as even moderately high risk. Indeed, over 85% of its investments are apparently at low or moderately low risk.

The GAO is skeptical saying that the DOD deliberately downplays delays and cost increases so as to reduce the
likelihood of scrutiny by the Office for Management and Budget (OMB):

“The DOD is masking significant investment risks, has not employed its own risk management guidance, and has not delivered the transparency intended by the Dashboard.”

The GAO’s recommendation to DOD? That more performance assessment information should be fed into the same process – in other words, more of the same.

Analyzing the GAO Report in More Depth

The report admits the limited usefulness of the dashboard:

“Both OMB and several agencies suggested caution in interpreting changing risk levels for investments … An increase in an investment’s risk level can sometimes indicate better management by the program … conversely, a decrease in an investment’s risk level may not indicate improved management if the data and analysis on which the CIO rating are based is incomplete, inconsistent, or outdated.”

So the implication is that none of the risk assessments can be taken at face value, and projects that are badly run are the most likely to have the most unrealistic assessment. But surely those are the ones most in need of effective risk management?

A Revolution in Risk Assessment is Needed: Agile Risk Management

The crux of my argument in my book “Agile Project Management for Government”, is that decisions should be based on practical feedback from what works – and that this feedback is needed early in a project lifecycle and frequently thereafter.³

My proposal in this blog is a simple one: we should not assess the risk of project failure by measuring compliance to bureaucratic assessments. We should instead rate all commitments to spend as ‘Red’ until they start to deliver, and then they should move into ‘Yellow’ status until substantial implementation has taken place that proves the project concept.

This will encourage smaller, more modular projects (as the OMB and GAO concur are required) and earlier ‘proof of the pudding’ from real-world success. This takes forward Barry Boehm’s theory that what he called spiral development where each iteration of a spiral of work starts with a risk assessment.⁴

The Federal IT Dashboard should be reworked divide money balanced on precarious assumptions about likely success of implementation from that being invested based on proven concepts that are being incrementally developed and released…

Comment below…

References

¹ US GAO “Information Technology Dashboard: Opportunities Exist to Improve Transparency and Oversight of Investment Risk at Select Agencies.” GAO-13-98, Accessed December 29, 2012, http://www.gao.gov/assets/650/649561.pdf

² Wernham, Brian. “Agile saves the FBI Sentinel project.” Accessed December 29, 2012, http://www.publicserviceevents.co.uk/ppt/mc12-brian_wernham.pdf

³ Wernham, Brian. Agile Project Management for Government. New York, London: Maitland and Strong, 2012

⁴ Boehm, Barry, and W. Hansen. The Spiral Model as a Tool for Evolutionary Acquisition.” CrossTalk, 2001

© Brian Wernham 2012 CC BY-NC-ND