Agile Document Development: Part 1 of 2: What is Agile?

The point of these posts: Agile methodologies used by software (and web) developers can be adapted to provide similar value to the rest of us. Not just for techies any more!

Introduction

This is a two part post. If you already know all about waterfall and agile development methodologies, feel free to skip this and wait for the next part.

This post grew out of a talk I was giving to our HR professionals on leveraging social media. I had prepared the presentation on that topic, sent in the slides for pre-loading and was ready to give it when I noticed something else on the session flyer. It had been promised that i would talk on “the accompanying culture of innovation and collaboration”. So I quickly gathered some ideas together for verbal presentation, which formed the nucleus of this post.

Fundamentally, I was thinking about an innovative and collaborative methodology for developing software in environments of rapid change and flux. It occurred to me that the principles behind this methodology would be applicable in other types of work that we do. What follows is my attempt to do so.

I’ll start by talking about what agile methodology is, contrasting it with the more traditional methodology used by the IT organization – the waterfall methodology. If you know all about agile and waterfall, feel free to skip this part.

In the next post, I’ll look at one of the most common tasks all of us face in the OPS, creating documents. Whether they are briefing notes, program proposals, strategic plans or policy papers, most of us create documents of some sort. It’s not for nothing that bureaucrats tend to be called “paper pushers”! I’ll talk about the similarities between how we develop these documents and the traditional IT waterfall methodology.

I’ll finish by outlining how the principles of agile development might be applied to our document development processes, helping to deliver “a culture of innovation and collaboration” that is perhaps more suited to our rapidly changing environment.

Software development – then and now

A. Waterfall methodology

I have clear memories of when I first heard of the Software Development Life Cycle. I was preparing to apply for my own job and I saw that arcane knowledge was part of the requirements. This was in the very late 90s or the beginning of the new millennium. There had been a major reorganization, with the creation of the current I&IT structure. My team had been moved from the business side of the ministry into the new I&IT organization. My position had been reclassified and the duties stretched a bit to be a bit more “IT” and knowledge of software development was now required.

What I learned, preparing for the interview, was a rigorous process designed to ensure that business requirements were understood and met, with the greatest possible efficiency and minimal risk. It went something like this:

1. I&IT meets with the business (usually a project manager and business analyst) to gather and understand the requirements for the IT system or solution being built. These requirements are documented in great detail and precision and are signed off by the business representative. If the solution delivers on the requirements, it is deemed to meet business needs.
2. The project manager and business analyst take the documented requirements back to the solution designer. The solution designer will look at the requirements and available resources and technologies and propose one or more options for a solution to meet the requirements. There may be different options based on balancing time, money and scope. These are brought to the business representative who will (select and) approve a recommended solution option for development.
3. The approved solution design goes to the developers for development. They write the code. They develop test plans and test the code in a variety of ways. (Does it do the things the requirements say it should? Does it hold up under load? Is it safe against attack? etc.) If it passes these tests, the business gets a chance to test it with “user acceptance testing”. If it fails any tests, it goes back for re-development until it passes.
4. Once it passes all tests it is deployed into production. People start to use it. Business needs are satisfied. Everyone is happy. The clouds part and the sun shines through. Somewhere, off-stage, a choir is heard. Way in the background, some people are maintaining the solution and others may eventually start thinking about potential upgrades.

That’s the waterfall methodology (more or less – it’s been a long time since that job interview). It looks great and, for many things, it works great.

But we started noticing that sometimes we weren’t getting the sunshine and choirs. There were a couple of reasons that might be the case. See if any of them sound familiar to you.

• IT gathers the requirements, gets approval and goes off to build and test. Six months or a year later they are back with a solution that meets the requirements, only to find that in the intervening time the needs of the business have changed. “That’s what we needed then; this is what we need now.”
• Again, IT builds the solution precisely to requirements. Business looks at the solution and says “Now that I actually see it, I realize it would work so much better if it were constructed this way instead of that way.”

To address these situations, a different methodology was developed – the agile methodology

B. Agile methodology

Agile methodology works differently. There are no long periods of development by IT separate from the business. There is no long-term lock-in of requirements. Here’s how it goes (more or less, there are different flavours).

1. IT meets with the business. They get a high level view of what the business is looking for, described as specific capabilities of the software. These are prioritized by the business. What they are looking for is the most useful capability/ies that they can build a working model of within a couple of weeks or so. Capabilities that can’t be built within that couple of weeks or so are preserved in the backlog.
2. IT goes off and builds it. This is called a sprint (aiming for speed over a short distance). At the end of the sprint, IT gets back together with the business and shows them what they have built. Only a couple of weeks development has been invested at this point. If the business says “Now we have seen it, we’d like these changes”, the changes are added to the backlog.
3. IT goes over the backlog with the business. The backlog will include things left over that didn’t make it into the first sprint, changes arising out of the sprint demo, and new priorities for the business. The backlog is re-prioritized and as many of the top priority capabilities or changes are selected as can be built in the next couple of weeks or so.
4. Repeat forever. After any sprint, the product can be shipped.

There are some risks and potential inefficiencies here. With waterfall, you know at the beginning of development everything the solution needs to do and how it will do it. With agile, things may need to be undone and re-done in different ways as new requirements emerge. Also, if you don’t have utility scalable infrastructure available, waterfall allows you to bring in the infrastructure as you need it (for development, user acceptance testing and production) and fully utilize it as it is brought on. Agile requires much more flexible infrastructure to avoid significant inefficiency.

But there are major benefits and risk mitigations to agile.

It is very much a “start small and scale” methodology. You are only committing development resources one sprint at a time. The end of every sprint is an off ramp with capabilities delivered to meet business needs. As business needs change, the solution changes right with it. Business and IT are working closely together, with constant business checks on the direction of IT progress. The minor inefficiencies or re-doing work as requirements change is much less than the major inefficiency of delivering a huge solution that no longer meets changed business requirements.

That’s all very nice you say (if you’ve read this far) but what does it have to do with documents? I’ll post about that shortly.