I’ve been wrestling with a thought lately – if organizations are complex systems, and complex systems are continuously self-organizing, then why do we believe formal processes make these complex systems more efficient? Worse, when an organization is in need, why do we engage in process improvement – when what may be needed is process reduction or elimination?
This is not the first paragraph to question process improvement, this is not some original Eureka moment. This is a personal journey, and the enormity of the mistake is beyond what I had considered previously. Friends, more erudite than I, have used similar words before – but for some reason I’m realizing, only recently, a simple truth: the implications for the baseless faith in the machine-based approach to management and the firm are global and profound.
A process-heavy enterprise isn’t cold and impersonal – because humans are still warm and social. Instead, a process-heavy enterprise creates the need for larger social networks. Formal processes do not capture the natural evolving paths people take to confront their tasks. In response, people do what is natural, they use their social network to navigate the workplace – looking inward to find others who have succeeded despite the process. We know that excessive time spent focused inward leads to burned-out employees, who must work the “second eight” to comply with organizational reporting and the like. On a larger scale, this wasted effort presents – at the limit – an opportunity cost for the enterprise as a whole. Perhaps the path to efficiency is to set the conditions for processes to emerge at the point of need, rather than Six Sigma-ing the (majority of) tasks that require creativity and agility.
In the famous early mistakes in business process re-engineering, managers believed once their processes were “streamlined” and “documented” (and embedded in enterprise software tools), they could realize savings by reducing the number of humans. For routinized tasks, this may be a reasonable assumption – however, what percentage of your workday is routine? Look to your own environment – do you rely on your social network to find the informal work-arounds for corporate process? When faced with a challenging problem, do you find solace in the documented process?
Work to Rule. In labor relations, there is a term called “work to rule.” Simply stated, this means that union workers have a negotiating tool that enables them to paralyze an enterprise – by merely doing only what is considered ‘by the book.’ No creativity, no work-arounds, no focus on task accomplishment – just fealty to the process. Consider this message: the way to crash some enterprises is to do what is expected by procedure manuals and process charts.
Business Development. In one company, I observed a set process for preparing contract proposals: with clear roles, authorities, assignments, formats, and process steps. Chokepoints were established along the way, when “pencils” down would precede a murder board review to assess the quality of the proposal against the procurement specifications. These comments were returned to the writing team, who would tackle their task anew. The information technology consisted of shared folders, and the writers laboring over each section would be required to post their documents in the appropriate folder at the required hour. The work was intense and draining, writers were often unaware of each other’s work, and the review team invariably excoriated the team for the lack of a “single voice” or “storyline.”
In another company, the proposal response was visible at all times to the entire proposal team. In a shared online space, the sections were worked in parallel, each writer able to observe the other’s ongoing work. The team met daily to talk through issues, but kept in touch throughout the process through instant messaging and email. There were roles and authorities, assignments and formats here as well – but the process was determined by the writing team, and emerged and adapted based on the demands of the work and the schedule. As the storyline evolved transparently, there were fewer surprises, people were able to lend value across the work throughout – and the end product was coherent and compelling. This without a review team’s intervention.
Software Development. In software development, Agile methods are triumphing over waterfall or other linear methods – users are happier because their approach to their work changes as they learn what is possible from the technology solution. The human and the software evolve together. The old approach was to gather what people thought they needed, build the software according to specifications, and then train the humans to operate the solution. There may be a correlation between how much training is needed and how disconnected the solution is from how people work. When software methods allow the humans and technology to co-evolve, when humans are co-designing the solution during “development” – we seem to have happier humans.
The thoughts bouncing in my head now are: what needs to be in place to allow for emergent processes? Formal process has a small place – compliance processes dictated by, for example, government regulation come to mind. However, value-creating processes must emerge from the interaction of the work and the humans. They cannot be formalized absent the humans or the situational context – if they are, then humans will circumvent them, creating a more inefficient enterprise… or follow them to the letter, and destroy value. In a real sense, process improvement should be replaced by process enablement. Let the approach to work emerge from the situational context.
Great post that reminds me about Perrow’s Normal Accidents theory. But I have to ask about the proof that agile software development methods are better than waterfall methods. I have yet to see one empirical study that demonstrates. Intuitively I feel that agile is better but hard proof would be great to have.
Reminds me a little bit of some of the IT debates that have happened for years on centralized IT vs decentralized IT. The hardest part is the truth is in the middle – people want clear structure with responsibilities and a path to success. But also some wiggle room and freedom. Finding this balance is the hardest thing for organizations
What would be your recommendations for someone wanting more Agile software development, but who is bound by government procurement rules and processes, not to mention organizational culture that tends towards “buy-in” from everyone (which can result in software being customized until it no longer functions)?
This is an excellent post concerning a serious problem, goes against the heavy process-centric orthodoxy especially in the federal government.
I have also been puzzled by your observation, “….Instead, a process-heavy enterprise creates the need for larger social networks…”. This seems to be true, I’m not sure it’s so much from the process heaviness as it is a lack of business function definition/guidance, and data understanding ownership? Everybody is involved with everything (stakeholders). Having grown-up in Bell Core telecom, there are “process heavy” organizations where these other factors were clear.
I have to sort of disagree about the proposed agile solution though, at least in the current state fed. If we had old school “staff” it might work, I’m talking software engineers that were staff just like the old days, that’s right I said it procurement people, a non “performance contract” or whatever you call it. I have seen agile methods work in the private sector, I have not experienced it working nor can I envision it working for a major non-trivial system (multiple major stakeholders) for either the contractor or the government (@Margaret).
Regarding Agile development, I’ll confess that I need to find something for Bill here. This isn’t much of an argument on the left coast, where I spent much of my time last year. A certain Fed CIO who arrived here from Silicon Valley recently decreed Agile methods be used, based on the success he’s had and seen in his career. And yes: this runs counter to the culture, procurement norms, staff experience. I owe Margaret a more complete answer to her thoughtful question below.
And Steve? I don’t see a balance here, friend. I see formalized processes as a significant barrier to value creation, innovation, and basic creativity. I have a friend engaging with me on Facebook who confessed she would take consistency over innovation and creativity – in her words, it “may be worth it.” Yes, for SCADA systems and nuclear power – I may agree with her at some level. But we apply formalized processes far beyond whatever value we believe they provide – I don’t see a balance, I see a need for them to be pruned far back. Let’s start with 80-20 as an opening bid. 🙂
“[T]he implications for the baseless faith in the machine-based approach to management and the firm are global and profound. Indeed! John, you are on to something I’ve studied in some depth. In fact, Routledge published my book on the subject a few months ago (The Political Economy of Bureaucracy). A common response to increasing complexity is further controls. Connectivity is great when it’s discretionary but – as the title you selected for this post suggests – spells death when it’s required. Complex organizations must enable ad hoc collaboration while being very careful that the ability to share information doesn’t create obligations that stifle innovation, action, and decision-making.