The three keys to innovation are to seek out new ideas, test these ideas on a scale where failure is survivable, and constantly monitor these trials for feedback. This is according to Tim Hartford’s new book, Adapt: Why Success Always Starts With Failure. Hartford argues that the world is too complex for top-down “big project” innovation based purely on expert judgment. The best path to innovation is to try a lot of ideas simultaneously (even if they contradict each other), build in robust feedback loops, and use the winning ideas to start a new round of trials.
This is not a new method of innovation; in fact it is the oldest method of innovation around – evolution. Nature is constantly creating variations of species and then selecting the species that best survive current conditions. What Hartford does is apply that concept to organizations to see if a similar process works in determining what companies succeed and which go out of business. The organizations that best survive a constantly changing business environment are the ones that combine incremental improvement along with the occasional long-shot idea to propel them into a better part of the business landscape ahead of their competitors
So, what does this have to do with government agencies? Hartford flatly states that this innovation method will not work in government agencies because of several barriers. First, there is not enough time for political appointees to fully see these experiments through before a new administration comes in office. Second, the process depends on a large number of failures for innovation but failure carries a high stigma in government. Third, it is difficult to clearly demonstrate that a policy innovation actually had an effect due to the lack of robust feedback loops in government.
I believe that Hartford’s opinion about government innovation is way overstated. There have been numerous government projects that have been extremely innovative: the Hoover Dam, rural electrification, the Interstate Highway System, the Moon landings, the Space Shuttle, the Internet, and so on. Even so, when you examine how these agencies developed these projects you do see that these agencies tried many ideas and learned from these trials. NASA has an amazing knowledge management culture and DARPA’s successful record of innovation is built on the concept of trying many long-shot ideas at once.
What holds government back from being even more innovative is the stigma of failure. Many agency cultures are too cautious because of the constant external scrutiny and the internal cultural practices of not sticking your neck out and just waiting out the latest change effort. In many cases, this caution is well-warranted. Many people depend on government agencies and thus agencies cannot fail in their primary mission of delivering Social Security checks, defending the nation, or enforcing laws and regulations
But failure to innovate will also lead to mission failure for agencies. In the sixth chapter of his book, Hartford describes how the 2008 economic meltdown was inevitable given the tight coupling of economic institutions and the failure of the government to prevent financial organizations from becoming too entangled. He argues that in any complex system, accidents will normally occur and that often our failure-prevention efforts will only increase the probability of failure. What is needed are the twin strategies of placing buffers between parts of the system and setting up feedback loops to warn us of emerging failure events.
Government has to constantly innovate so that it can continually deliver on its mission. This means that the culture has to change so that the agencies accept the small failures that teach to avoid the large failures that cripple the agency and harm the people it serves. Whether we call it “experimentation”, “pilot tests”, or some other euphemism, the better government is at innovation the better it can serve its citizens.
Disclaimer: The opinions in this posting are solely mine and do not reflect the views and opinions of my employers or any organizations I belong to.
References:
Belfiore, M. (2009). The Department of Mad Scientists: How DARPA Is Remaking Our World, from the Internet to Artificial Limbs. Smithsonian.
Hartford, T. (2011). Adapt: Why Success Always Starts with Failure. Farrar, Straus and Giroux.
Insightful post, Bill.
In practice, has anyone seen examples of healthy competition between offices within a single government agency? I don’t work inside government, so I recognize that there may be lots of examples that I haven’t heard about.
This question comes to mind after reading one particular statement you wrote:
I have a hard time imagining that there are many managers in government (and for that matter the private and non-profit sectors) who have two teams of subordinates simultaneously working on competing, mutually exclusive proposals in isolation from each other. I’m guessing that would seem threatening in all but the most innovative cultures. Anyone ever seen it done inside or outside government?
@Lucas – Thank you for your comment. I think DARPA has had healthy competitions on pilot projects but I am not aware of any government agency that has done this. I have seen examples of unhealthy competition which is what prompted my thinking on the subject.
I am hopeful that the current approach to challenges would lead to internal healthy competitions between offices. One reason for my post was to encourage more pilot projects within government agencies rather than another “all the eggs in one basket” big project.
Nice post, Bill. Your post reminded of a Bob Dylan quote – “There is no success like failure, and failure is no success at all.” I’ve heard similar observations about the stigma of failure as a barrier to innovation in government. Other comments I have heard is that innovation in the public sector traditionally comes from 1) a crisis and 2) a “champion” in the organization who can successfully navigate the political arena to push ideas forward. Any comments from your experience? Hartford’s book sounds interesting too, I might have to pick it up, Thanks!
I would like to collaborate better. However when posters don’t identify their organizations, use abbreviations and terms with which I am unfamiliar like ning, I cannot. This is exaggerated by my visual and dexterity disabilities. Do you think having a style manual govloop sends out as individuals sign up would help?
I think we need to keep in mind one of the major reasons government failures attract so much stigma is they are often NOT survivable (failure of levees during Katrina, collapse of Teton Dam, casualties from wars based on faulty intelligence, etc). Also the cost and damage of government failures scales with the size of the project and can go off the charts. Look at the cost of the war on drugs and the war on poverty, both of which would have to be described as failures. Finally, a private sector corporation whcih fails is answerable to its investors, most of whom knew (or should have known) the risks they were taking. Governments are answerable to voters who are often unaware they are assuming any risk at all until the failure of X project jacks up their taxes or devestates their community or both. Failure is a necessary evil which must be accepted in order to make any real progress but I’ve observed that leaders who become too eager to embrace failure in an effort to demonstrate their cutting edge managment philosophy tend to acheive failure with a disturbing regularity.
@Peter – Great points. I especially agree with your last sentence. That is why Hartford argues for pilot projects where failure is survivable versus big projects where failure is catastrophic. In chapter six, he addresses the issue of projects where failure cannot be tolerated. The paradox here is that by avoiding failure, the projects often end up more likely to fail as Perrow (“normal accidents”) and Reason have written about.
@Carol – It would be great if GovLoop had a style book but I think the ultimate responsibility for readability does fall upon the author. I am not aware of any accessibility features in Ning but I am certain GovLoop does its best in 508 compliance.
I would argue that innovation is held back within government due to the focus of many government agencies. Instead of focussing on the traditional “3 P’s” such as people, product or profit it has a prediliction to focuss on process. Government institutions need to change from being beaurocratic and process driven into being innovative and citizen focussed. I would argue to do this requires significant cultural shifts – which do allow experimentation, where collaboration is common and the direction is very clear
@Christo – I agree with the need for changing culture and more focus on people/product. But profit doesn’t apply to government agencies. I would suggest that processes are important but there should be a balance between processes, people, and product. Great idea on the 3 P’s.
Like @Lucas, I think the key here is to see some case studies. What are the specific examples of DARPA using innovative, competitive approaches to problem-solving and project management?
I also agree with @Pat that most innovation happens when there is top cover and assurance that someone will not lose their job when innovation and failure happens.
Another consideration: we need to redefine the term “failure.” Right now, failure is happening across agencies. That’s inevitable. We aren’t going to get it right all the time. But it might not be defined in terms of “doing things the way we’ve always done them.” That’s likely considered success – stability, status quo. For the kind of failure you’re advocating, we need to examine old ways of doing things and ask, “Is this still the best way? Are there novel approaches that we haven’t tried (or tried before with different people)? How do we test and see?’
@Andy: Belfiore’s book is an in-depth examination of how DARPA uses innovative and competitive approaches to problem solving and project management but the main lessons are to constantly rotate program managers so that they don’t become set in their ways and to flatten the layers of the organization so people can more easily communicate their ideas to top management. This approach has led to an amazing failure rate of 85% to 90% in that many proposed projects don’t fully meet expectations. Some noteworthy examples of failed projects:
1. Mechanical elephant – developed during the Vietnam War to help soldiers travel through the jungle terrain.Kind of like the AT-AT Walkers in “The Empire Strikes Back” but just didn’t work.
2. Telepathic spies – If you are old enough to remember the Donahue episodes where he alleged psychics that could “remote view” Soviet military bases. Just didn’t work.
3. FutureMap – using a public prediction market to determine when the next terrorism strike would occur. Public outcry over betting on terrorist attacks killed this idea.
But, consider the 10% of the projects that did work:
1. The Internet – Imagine today’s world without it.
2. GPS – Almost considered a necessity today.
3. Stealth technology – Consider how vital this is to our nation’s defense. (More examples can be found in this article from New Scientist.
Even with such a high-failure rate, DARPA’s successes more than makes up for the 90% of projects that fail. As Hartford argues in his book, it is impossible to calculate the return on investment (ROI) of highly-innovative projects that do succeed. What exactly is the ROI of the Internet?
This is why you see agencies like the Department of Energy and the Department of Education starting their own DARPA-like programs: ARPA-E and ARPA-ED respectively. And we do know of one agency head, Martha Johnson at GSA, who publicly advocates failure as part of the innovation process and urges leaders to stand by their employees who innovate.
“As a leader, it’s very important to have a real clear eye about risk and to be very intentional and transparent about it. I think we need to learn to fail. I have a little saying, fail forward, fail fast and fail fruitfully, because you never learn or innovate if you always do it right. I think leaders have got to set a different value structure to their people, and say, ‘I will stand by you.'” (Washington Post, August 3, 2010).
As to your last paragraph: I fully agree we need to redefine both failure and success. We need to constantly re-examine what we are doing and determine if we need to keep doing it this way or if we even need to do it all.
Love the topic, here’s some things I’ve found on the subject:
Love this: “constantly rotate program managers so that they don’t become set in their ways and to flatten the layers of the organization so people can more easily communicate their ideas to top management.”
And not just within agencies but across agencies. We need to create something akin to the PMF program for senior leaders and program managers in government to bring fresh perspectives to long-term challenges.
@T. Jay – The Spitfire story is great and rather entertaining considering the personalities involved. I need to pick up a copy of Little Bets. And I love the picture on your blog post. I suppose we all were overwhelmed by the recent royal wedding.
@Andy – If you like that, then you will love Chapter 3 of the DARPA book which details how the Internet was created. It was essentially a 20-minute conversation that a DARPA employee had with the director after the employee had an “Aha moment.” (p. 71)
If appears to me that we Government gets most things right. We win most of the wars, terrorist attacks were thwarted, social security mostly gets paid to the right people, etc. However because leadership believes that it must be successful it sets small rather than reaching goals for itself. Imagine how much more exciting work would be if we were challenged, or allowed to miss marks! Government is waaaay too cautious.
I believe that government would be better lead if determined who were the leaders of CHARACTER and competency and empowered then rather than just rotating smart people into positions. It’s not IQ its emotional health, maturity, integrity, commitment, courage, etc. Even in “dumb dog” packs they only follow the alpha dog because they trust him to sniff out the deer, organize the hunt, and fight off the wolves, or take care of them. When he can’t manage that mission he is challenged and replaced by the more competent.
FYI-When researching this it I learned that only humans WANT to be dominant for the mere purpose of it. But then canines don’t get promoted or cash awards.