The point of this post: Just as the web changed our model of content publishing, so it will change our model of application development. Government IT shops need to evolve to keep up.
Publishing and the Web
Once upon a time, a writer would write a book (and re-write it and re-write it). Then a publisher would take the finished product, set it irrevocably in type, print a lot of copies and distribute them. The book was a finished product in fixed form and everyone accepted that. Sure, new editions might come out – but a published book wasn’t a “work in progress”.
The web changed the way, not only that we publish, but that we approach publishing. Unlike a book, a website is never a finished product. That’s why we never see those oh-so-cool animated “under construction” graphics any more. They were pointless. Everyone knew everything was always under construction. Some changes are bigger (a new “look and feel”, content reorganization) some changes are smaller (content added, changed, removed), but every day the site is changing. That’s a fundamentally changed model.
Publishing and Software Development
The way we get software has, to some extent, modeled the way we get books. Software developers would write an application (and de-bug and de-bug it). Then a software development company would take the finished product and print it onto floppies (or CD-ROMS or DVDs) and distribute them. The software was a finished product in fixed form and everyone accepted that. Sure, new versions might come out – but published software wasn’t a “work in progress”.
It wasn’t substantially different for non-commercial software developed in-house and published to servers rather than distributed by disk. You’d gather requirements, design and develop, test and publish. Then you had a fixed and finished product that could go into maintenance mode. You may at some point gather new requirements and start over, ending in a new release. But that’s like a new edition of a book. It doesn’t detract from the “finished” nature of the previous release.
Software Development and the Web
Just as with the publishing, the web is changing all of that. Emerging web technologies are moving the web away from being just a publishing medium to being an application medium, too. But the “always changing, never finished” nature is persisting.
All of the big social media websites: Facebook, YouTube, Flickr, Wikipedia, even Google – they are always changing. I was at a talk yesterday where the speaker said that one of them (I think it was Flickr) makes hundreds of small changes a week to the application. And everyone knows how often Facebook changes (especially its privacy settings!). These changes aren’t presented in the formal “versioned” manner that traditional software changes are. No one could tell you what “version” of Google we’re on. They’re presented as the evolution of a website constantly subjected to tweaking and improvement.
Application changes may be slipped in as unnoticeable (or barely noticeable) improvements. New functionality may show up as “beta” or “Labs” with the explicit understanding that it is rough and will rapidly undergo significant changes.
Changing Expectations
That’s changing the expectations that people are bringing to their government IT departments. Like the applications they use online, our users are expecting our applications to nimbly respond to evolving needs immediately after they are released. People seem to be increasingly responding with “Why are you holding me to requirements I gave you a year ago when I hadn’t seen the product” when we deliver the carefully wrought results of our waterfall methodology.
The Production Prototype
The speaker I was listening to yesterday was talking about the design process. Unlike the waterfall process (as I’ve generally seen it performed) which places application design completely before application build, the design process he described places the build squarely in the middle of the design phase, with the iterative creation of functional prototypes. These quick, cheap builds are evolved, though testing with users, before design is completed and they go into getting whatever widget they are designing ready for mass production.
It struck me that websites (and web-based applications) never go into mass production. There’s no need to ever leave the prototyping stage. They may go into “production”, but they continue to be tweaked and improved like any prototype.
Increasingly, this is being demanded of other applications, too. (Especially as more and more of these are being delivered through a web interface.)
This can be kind of scary for a more traditional IT shop. “You’re telling me that I should be using prototypes to run mission critical applications!? How can I support that? How can I ensure stability, availability, reliability, etc.?”
I can’t say I have all of the answers. I can say that Google, Facebook, Flickr and YouTube seem to be fairly reliable. They may have had outages, but I’d put their availability track record up against the internal “mission critical” apps I use any day. So I’m pretty sure that someone has figured out the answers. Maybe we can learn from them.
What do you think?
This is how 37 Signals became so successful. Put out a simple application and let users work with it. They will suggest new features that can be easily added in real time because modern web applications are built by using loosely-coupled modules that work off a common API (application programming interface). This evolves a robust application from the ground up that has the advantage of being tested continuously as it is developed.
This contrasts with the traditional style of application development as you point out in your insightful posting. You might want to read Matthew Burton’s “A Peace Corps for Programmers” (in Open Government: Collaboration, Transparency, and Participation in Practice) as he has similar arguments plus a solution to bring about the new way of building applications for government.
One of the challenges IT shops have always faced is that programs who want new technology to fix a business need have 2 choices – 1) Do the official thing, go to IT for help 2) Ignore IT and rules and go buy and maintain themselves.
There are a lot of problems when business units go with #2 especially when staff change, requirements change, and security issues.
But if #1 is very slow, cumbersome, and way too expensive….IT has a huge competitor. It’s called business units going outside official channels.
There is a 3rd choice as someone pointed out below -Software as a Service – where the core app based on common need of the market is already offered on platter to the customers. 37signals did it quite well with their products, whereas SAP which still is going strong is much much much costly and rather inefficient model.
Additional features, customizations can go on forever over the core infrastructure as the customers mature by using the tool. And the core functionality is maintained by the company behind SAAS app, which ensures years of performance and cost-effective upgrades.
Very nicely raised question David. The obvious answers to future lie in SAAS, opensource and other such cost-effective, efficient model.
cheers
Arvind
CEO http://bubbleideas.com
@Arvind – Good points in your posting. SAAS is a good way to go but what makes 37signals work better than SAP is that a developer can use the API to extend 37signals products and have those products work with other products. It’s an open system whereas SAP is not as easy to extend because they want you to stay in their closed system.
Another example is Drupal. You have a core set of process for building a good content management system and then you can easily extend the core platform with user-contributed modules or build your own.
It’s all about lock-in. Big vendors know that they can make a great deal of money by selling closed systems to government agencies by locking the agency into a platform that the agency has to depend on the vendor to manage and upgrade. When I worked at a DC political organization (name withheld on purpose), I replaced the former developer who used a niche project to create their membership management system. No one else knew how to program the system and whenever the organization needed a new form for data entry or a new report template, they had to go back to the developer who charged a great deal for his services.
I grew tired of this and recreated the system in Access 95 so that we could easily create forms, reports, and data entry screens without going through the developer. I was also able to use macros and Visual Basic to create some custom modules.
I would like to say that my efforts were a success but organizational inertia convinced the top management to stick with the former developer. I fear that the same organizational inertia will block the adoption of SAAS despite the best efforts of GSA to push their cloud platform with associated apps.
@Arvind: SAAS is certainly one possibility. However, there may be times when for one reason or another you really want to create something in-house. Why can’t our internal IT providers use those techniques that allow SAAS providers to combine nimble adaptability with reliability and availability?
@David – You might be interested in this slide presentation: How Darwin’s Finches and API’s Are Connected in the Post 20th Century Economy.