Phase 2: Evaluating Alternatives

In this post we will extend our product lifecycle, adding detail that creates epics from themes, considering buy versus build, and cost. Finally, we will evaluate the options using four criteria.

The first step requires business analysis: we take each of the Themes from Phase 1 and decompose each into 5-10 Epics. Again, if you get less than 5 Epics or more than 10, then you are likely decomposing into too little into too much detail. Like the Themes, each Epic should be associated with one or more business outcomes.

For each feasible option, you need to estimate a rough set of costs: upfront license costs, annual support costs, the frequency of license upgrade renewal costs, customization/development costs, integrations costs, deployment costs, and operational costs.

The, from this, you engage in market research to determine if there are commercial off-the-shelf (COTS) products that seem to satisfy the epics and outcomes. Products should only be included that are genuinely off-the-shelf. COTS products that require modification (MOTS) will be considered next.

There is a general truth to evaluating buy versus build that is worth stating. The more you can customize the solution to your needs the more satisfied your users will be. Historically, the more you customize your solution, the more cost, and risk you assumed. I believe that this historical statement becomes less accurate every year. Let’s look at build options to see why: MOTS, Low-Code, and custom development.

MOTS products require the vendor to branch their core software to satisfy your Epics/Outcomes. They can be risky to develop, expensive to develop, and expensive to support. Nonetheless, the risks and costs may be less than other options.

There is a new buzzword in the space: low-code software. These products are more easily extended and customized than MOTS, significantly reducing the costs of customization over MOTS options. Still, there is code to write, test, and support. There are license and support costs. The low-code framework is not wholly extensible so that you will live within the framework.

The way Silicon Valley companies develop complex applications fast and quick is through the reuse of open source code. Reuse means looking at GitHub or SourceForge for public code snippets, modules, or applications that solve for all or part of your epics and taking a fork of the code as your own. This borrow versus build approach is the single largest accelerator of developer productivity today. It is normal to find open source modules that solve 20%-60% of many application requirements (if you look into the COTS, MOTS, and Low-Code codebase you will find that they often have borrowed open source code to extend their products).

One last comment on this Buy vs. Build decision. Because “Borrow” is such a compelling option you rarely see start-up companies requiring large sums of money to get an initial software product to market. I have been told that Facebook spent less than $50M on IT before they went public with several hundred million concurrent users. I have been told that Salesforce spent even less than that before they were live. The point is that when you cost Borrow options for large applications they should come in at the $10M-$40M mark. Today there are still lots of $200M-$400M software development projects in the government space. I believe this is due to an old-school mindset that is not based on modern engineering best practices and current borrowing of open source code. There is just no such thing as a $300M venture-backed software company. More on this below.

Here is a simple way to quickly evaluate the three options:

• Rate the fit of the alternative to your requirements on a scale of 1-10. A Build should get 10, almost by definition. If your development team is not very responsive to user requirements, it might get less. COTS will rarely get 10 points and may get less than 5. Don’t fret as COTS will make up the points later. MOTS may get 1 or 2 points more than COTS. Low-code might get 1 or 2 points more than MOTS, but less than a custom build.
• Estimate costs starting with COTS. Be sure to include the installation and integration costs. MOTS will cost more than COTS as you require a commercial license, installation and integration, and modification. Low-code will be about the same as MOTS, or likely a little less as the cost of modifying low-code is less than the cost of forking a COTS product. Build could be anywhere, high or low. However, if your build estimate is more than $30M then something is wrong (see the comment on the cost of software start-ups). Note that build estimates will double or triple if you force development vendors to a fixed bid. Startups spend private dollars which is like time and materials bids.
• Evaluate the total cost of ownership. If you have to buy hardware and a network and operate it, add it in. If you have to pay ongoing license fees for support, add it in. For Software as a Service (SaaS), license and infrastructure are baked in. If you have a COTS or MOTS alternative, add in 15%-20% for ongoing customization as requirements change over time. If you have a low-code alternative add in 5%-10% for customization. For a Build alternative add in 10%-15%. For each alternative include funds for system administration.
• Evaluate risk. COTS is very low risk. MOTS is a very high risk (remember, if any code in the COTS changes it is MOTS). Low-code is in-between COTS and MOTS. If lots of low-code is required, then the risk is high. A custom build is in-between. If there is very little code borrowed then the risk is higher than any other option. As an estimate, assume 30%-40% of the code is borrowed.

Here is a reality check for old-timers. If we assume that 33% of the code is borrowed and the cost of the build is at or below my $30M target, then in the old days the cost would be $45M if all of the code was custom. If the contract is for a fixed bid, then the $45M cost goes to $135M. If the ongoing support costs for a license, infrastructure, and further customization is 35% per year, and the contract is for 4 years then in the old days the cost would be expected to be $240M over 4 years. Does that sound familiar? A custom build, time and materials with 35% per year could be $60M. They could be less if infrastructure costs are at cloud prices.

So there are several points made here. There are four basic approaches to be considered: COTS, MOTS, Low-code, and custom build. It is noted that today a “build” is actually a “borrow and build.” Four evaluation criteria are listed with some guidelines for how these evaluations might be expected to work out. Your mileage will vary. We did a reality check to explain why programs were big and expensive in the past and to show why they may not be so expensive today.

I hope this was interesting. Next time we will consider how to develop a recommendation into an approach and to generate a statement of work.

Rob Klopp is part of the GovLoop Featured Contributor program, where we feature articles by government voices from all across the country (and world!). To see more Featured Contributor posts, click here.