How to estimate and measure software quality costs
Software quality is the investment an organization makes to prevent the risks associated with defects in its software product. Any investment is measured through return on investment (ROI), when the gain from the investment is compared to the cost of that investment.
Cost of investment is, in most cases, easy to calculate – it’s how much your organization pays for QA services (whether performed by a third-party vendor, internal QA engineers, or other team members such as developers, business analysts, project managers, or business representatives).
In some cases, there are hidden costs as well, as in the following scenario: an organization has no dedicated QA resources, and all testing is expected to be done by its marketing department. The marketing department assigns one or several representatives who spend more and more time doing testing instead of marketing-related activities. The marketing department’s capacity decreases, which leads the company to hire additional marketing resources. (Unfortunately, it’s very common for organizations to hire a marketing specialist rather than accept the fact that they need a QA resource.) That results in additional payroll, benefits, and human resources costs.
The gain of such an investment is not obvious and depends on the nature of the software an organization is building. Let me give you a few examples.
1. Invest now to save in the future.
Steve McConnel, in his book Code Complete (1993), described the cost of a defect discovered in the early stages of the software development lifecycle as opposed to the later stages. He explained that the cost of fixing a defect when software is already live is much higher than the cost when it is in the requirements stage. Why? Your team has already spent time (and money) to develop and implement software and then has to make changes – in requirements, codebase, database, integration points, user guides, and elsewhere.
Are you willing to invest in quality assurance in the early stages of your project or wait for a snowflake to grow to the size of a snowman?
2. The customer is your first priority.
Have you noticed how customer service is improving for bigger companies like Amazon or Apple? They do their best to make sure your experience is great even if you don’t like the product and end up shipping it back. The reason for that is pretty simple: customer acquisition cost (CAC). Every company that wants to stay in business for more than one holiday season is thinking strategically: they’d rather lose some money now (and take care of free return expenses) than lose a customer who is willing to pay.
That’s very true for any kind of public software (like Facebook, Uber, or Amazon retail). The company knows that if you shop once and enjoy the experience, you’ll come back for more. That means you’ll be spending money (or watching ads), which means the company’s revenue will grow. How often do you see those “Sign up and get 20 percent off of your first order” deals? They’re everywhere.
Or look at how smart Apple’s model is. they start by selling you an iPhone, and the next day you also have Apple TV, Apple Music, an iPad for your mom, an iPod for your niece, and an Apple Watch as a Christmas present for yourself. Oh, and AirPods, chargers (one set for work, one set for home, and another for travel), a cute pink case, and a new watch band. The list is endless. Once you’re a happy customer, you’re a customer for life (with some restrictions, of course).
However, as I’ve mentioned above – easy come, easy go. Low-quality software products will sooner or later push your customers to look for alternatives. And they will find them, while you’ll have to start from scratch. The CAC will go up and up and up, until you see that the CAC is much too high in comparison to revenue growth for your organization to stay profitable. Do you remember Windows-based smartphones? Right, no one does.
3. Identify the direct cost of errors.
This one is very easy to measure when an error occurs. Have you heard the recent story about a Bank of America ATM that mistakenly dispensed $100 bills instead of $10 bills? Bank of America decided to allow its customers to keep the money (a nice gesture to please customers, by the way), but the company definitely knows how much it lost.
I believe that was a human error that occurred when filling the ATM, this is exactly what happens with software defects.
Wrong discount calculations for Black Friday? Super cheap flights? Accounting software that rounds numbers incorrectly? If there is an error in your software, you will lose money. You’ll be lucky to find the root cause and fix it, or you may have to live with the error forever because people want to believe all is good.
In a nutshell, quality does not help you gain money directly, but it does protect you and your organization from additional expenses – CAC, direct losses, reputation loss, and (let’s hope it never happens to you!) liability expenses. We all know how much those can cost your organization.
How to calculate ROI?
Calculating ROI is not an easy task and may require some statistical data that is not always available. I recommend introducing Quality Metrics – a set of criteria to keep an eye on how QA is (or is not) helping your organization. This set of metrics is normally defined together with a client and then organized for semi-automated calculation (thank goodness for project management software!).
A few of the most popular metrics are Defect Density and Defect Cost.
Defect Density allows one to see how many defects you have in an application in a test case or test scenario. It’s important to track this one in time, as you’d expect the number to decrease the closer you are to the end of the project.
I also like Defect Cost to track how much effort (and, thus, money) you spend finding one defect in the solution. This is a good metric for those comparing several QA providers.
All in all, our full list of metrics includes about 15. For a specific project, we’ll probably select five to eight. But measuring is just the tip of the iceberg. Please, please analyze what you see. Only a thorough analysis can help make your QA situation better than before.