Steve would like to thank colleague Kevin Haas for his helpful comments.
In October of last year, I received an email from a prospect in the IT department of a small company looking to implement an open source business intelligence (BI) application. Included in the note was a side request to help address the return on investment (ROI) of the proposed solution. When we followed up over the phone, ROI was at the front of the discussion. The prospect told me that before he could get serious about the initiative, he had to demonstrate the project's value to the CFO. He wondered if I had a spreadsheet that could provide the foundation for the calculations he needed.
I did have a spreadsheet that could help him tabulate the cost of deployment, but it, unfortunately, provided little guidance on quantifying benefits. And, despite being able to provide at least some assistance, the interaction raised several red flags for me. What was the companys motivation for BI? Was the business side involved in the effort? Or, was the BI program primarily an IT initiative? How would the company articulate and quantify the business benefits? Like it or not, I was back in the BI ROI business.
In the mid to late 1990s when technology was king, I recall assisting customers with their ROI calculations. At that time, the ROI determination effort often seemed pro forma, almost as if it were a meaningless checklist item. Supported by their ROI analyses, companies were investing heavily in BI. Life was good.
In 2008, there is no shortage of opinion on BI ROI in the thought leadership market. Hardware and software vendors regularly report analyses demonstrating the positive return on their BI products. Go figure. A variant of this vendor-driven ROI review is a product evaluation by a recognized expert or survey firm focused on cost-benefit. Such reviews are generally more thoughtful, using currently accepted BI concepts to frame product ROI considerations. In fact, I've read more than one of these articles just to keep up with the celebrated authors before I even realized they were shilling for a product. While expert ROI reviews can certainly be useful, readers must be mindful that findings are preordained to a positive spin on a product.
Surveys of BI users fully demonstrate the difficulties of quantifying the hard benefits of BI. Many practitioners are content to simply contrast the performance of their BI solutions against going in expectations. Some observe soft benefits such as quicker and more accurate reporting and richer support for decision making. Metrics noted include both an increase in access to information and a decrease in the time of that access. Enabling more users with easier access is cited as well.
There are articles that tout a pure financial calculation approach to ROI, using precise mathematical formulas to specify the costs and benefits of BI. There's certainly appeal from the CFO perspective for such exercises. The problem, however, is that inputs to the formulas are often imprecise, and sometimes arbitrary estimations. Benefit projections three, four, five or more years out are risky at best and spreadsheet jockeys can readily tweak the inputs to build their case. Alas, this approach might be more the rule than exception.
My own philosophy has been to help customers identify the cost side upfront, then make full cost-benefit analysis a topic of a roadmap engagement that kick-starts the BI initiative. The key activity here is a high-level analysis conducted with a performance management lens that links business strategy to measurement through a series of if/then hypotheses. This method both engages business with IT and provides the inputs to solidify an understanding of program ROI. The approach is not unlike the Business-Centric BI Methods, espoused by Nancy Williams and Steve Williams in their excellent August 2004, DM Review Magazine article.
Of course, the performance-driven roadmap still requires customers to make a leap of BI faith to devote the time, energy and dollars to conducting the ROI study. Unfortunately, as BI blogger David Loshin commented, companies often will not make this commitment to identifying the benefits of BI. It seems Loshin suffered the same fate with his customer as I did with mine - a process that stalled without the requisite IT/business collaboration to determine ROI.
Those who share the frustration with the current state of BI ROI can find solace in an excellent Harvard Business Review (HBR) article by professors Clayton M. Christensen, Stephen P. Kaufman, and Willy C. Shah entitled Innovation Killers - How Financial Tools Destroy Your Capacity to Do New Things. As the title suggests, the authors are not big fans of the current obsession with single-minded financial analysis to determine the fate of new business initiatives.
For Christensen, et. al., a new business innovation like BI is often disadvantaged out of the gate by traditional discounted cash flow (DCF) and net present value (NPV) calculations. The problem is that the present value of business with BI is contrasted with a business without BI that is assumed to have unchanging performance. In fact, the authors argue, the do-nothing scenario more often results in a decline in performance over time, brought on by price and margin pressures from changes by the competition. This Parmenides' fallacy is further exacerbated by errors of estimation in the cash flows for out years, three or more years in the future, which compound mistakes in initial assumptions. And for Christensen, et. al., the out years are generally when the decline in performance accelerates in the do-nothing scenario.
In addition to the potentially pernicious financial calculations, the authors discuss internal review processes that can also kill innovation like the adoption of BI. Stage-gate review starts with a portfolio of potential new programs progressively culling the possibilities through a series of phases until only the strongest remain. The review process generally involves three stages: feasibility, development and launch that are separated by the gates of review meetings. Review criteria are often anticipated revenues, profits and risks - considerations that favor incremental rather than disruptive change in the battle for scarce funding. Of course, it's easy to tweak assumptions and inputs to get a proposal by a gate. Destructive change, which could well include new BI initiatives, is also at a relative disadvantage because its strategy cannot be completely known upfront and therefore runs the risk of being stopped at future gates. Stage-gate review is thus conservative, favoring incremental rather than disruptive change.
The authors, fortunately, offer the prospect of a review solution that supports growth investments like BI. Discovery-driven planning reverses the premise of stage-gate review, first constructing acceptable revenue and cash flow, then identifying the inputs necessary to make the numbers fly. Project teams create a rank-ordered checklist of critical assumptions that can be reviewed and tested continuously. As projects enter a new stage, the checklist is revised and used as a plan, with the intention of quickly testing and learning whether inputs are still valid. If, as the effort progresses, assumptions prove untenable, the project is scuttled. With discovery-driven planning, the ongoing emphases are the assumptions, which can change with learning, not the numbers, which are already credible. As Christensen, et. al. note: More often than not, failure in innovation is rooted in not having asked an important question, rather than in having arrived at an incorrect answer.
At the end of the day, we're left with only slightly more than confirmation of our frustration with the state of BI ROI. Our misery, at least, has the company of a highly-regarded publication like HBR for support. All is not gloom, however. What is needed is a different lens to shed new light on the ROI concern. What if, right from the start, all business processes routinely use intelligence and analytics to evaluate performance? What if the knowledge driven by BI is seen as being as central to running the business as the processing of transactions in the enterprise resource planning (ERP) platform? Would company leaders then be quite as concerned with a go/no-go BI decision based on simple DCF and NPV? We think not. We think that BI and analytics are beginning to emerge as every bit the indispensable apps as ERP. An upcoming OpenBI Forum will explore that thinking, looking to the HBR for guidance.
- Clayton M. Christensen, Stephen P. Kaufman and Willy C. Shih. Innovation Killers: How Financial Tools Destroy Your Capacity to Do New Things. Harvard Business Review, January 2008.
- Nancy Williams and Steve Williams. Capturing ROI through Business-Centric BI Development Methods. DM Review Magazine, August 2004.