Who’s to blame?
Companies like Amazon, Google, and Microsoft rely on large, complex systems to provide services to their customers. When something goes wrong with these systems, they urgently want to understand why so they can prevent a recurrence. Government agencies also rely on complex systems to provide many vital public services. When something goes wrong with these systems, we often want to identify who’s to blame and hold them accountable.
For much of the past decade, government agencies have been adopting private-sector software development practices to improve the development and delivery of their digital services. They should also consider adopting an increasingly common practice used in private industry: the blameless postmortem.
What’s a blameless postmortem?
A blameless postmortem (sometimes called a blameless retrospective or review) is a process organizations use to analyze and learn from incidents, outages, or failures. The primary goal of this process is to understand why something happened, not whose fault it was. This approach promotes a culture of learning from mistakes to avoid making the same ones in the future.
The “blameless” framing of these processes is important: It encourages people to raise issues they suspect may lead to a failure or share details to help identify the root cause of a problem. Focusing solely on who to blame in the wake of an incident can create disincentives to sharing important information. If people fear consequences for raising issues or talking about mistakes, they are much less likely to do so.
Developing a blameless culture
Adopting blameless postmortems is part of a larger cultural shift in moving away from assigning blame to one of continual learning. This shift can be difficult and often takes time. When complex systems fail, particularly systems managed by government, people may suffer harm. In these instances, our desire to assign blame may conflict with the goal of identifying the root cause.
In 2014, the actor Dennis Quaid and his family were impacted by a failure in a complex system. The family’s newborn twins, while in the hospital following their birth, were given a massive overdose (1,000 times higher than what was prescribed) of the drug Heparin, an anticoagulant. This created a potentially serious risk to the newborns.
The process for ordering, preparing, delivering, and administering drugs in a hospital is a complex one involving multiple steps and many people including doctors, nurses, and pharmacists. Checks exist in this process to ensure that only the proper dosages of the correct drugs are administered. But these checks didn’t work in this instance. The system failed, and people — understandably, given that small children were involved — wanted to know who was to blame.
The root cause of this issue, it turns out, had less to do with individual negligence and more to do with a design flaw in the system. At the time, vials of Heparin at wildly different dosages used similar labeling, often leading to the administration of incorrect doses. At a glance, a vial of Heparin at a concentration of 10 units/mL might look similar to one dosed at 1,000 units/mL: Both labels have a similar background color, layout, and font. Even for diligent doctors and nurses, these different dosages could be easy to confuse.
In an attempt to solve this problem, as later prescribed to drug manufacturers by the FDA, the manufacturer changed the labeling on Heparin vials to make them easier to differentiate.
Understanding the root cause of complex system failures can be tricky, and our desire for accountability may sometimes be at odds with learning it. Incorporating blameless postmortems to help identify issues in their own systems, could help government agencies provide better services to those who need them.
Mark Headd is a Government Technology SME at Ad Hoc. He is the former Chief Data Officer for Philadelphia, serving as one of the first municipal chief data officers in the United States. He holds a Master’s Degree in Public Administration from the Maxwell School at Syracuse University, and is a former adjunct instructor at the University of Delaware’s Institute for Public Administration. He spent 6 years with the General Service Administration’s Technology Transformation Services (TTS), serving on the leadership team for 18F and leading customer success efforts for TTS’ cloud platform, which supports over 30 critical federal agency systems.
Leave a Reply
You must be logged in to post a comment.