It’s tempting to measure cost alone because quantifying the value from competing BI solutions is so difficult. But value is too important to ignore completely. Even if you can’t measure it directly, it’s worth finding an indirect approach.
One method is to consider time. Even if we don’t know what an answer is worth, we are pretty certain its value is higher if we get it sooner. Phrases like “time to answer” or “time to insight” clarify that they represent an end-to-end measure rather than individual components such as response time.
This is important for the new technologies because many of their advantages come from eliminating noncomputer tasks such as design time or meetings. These are not captured in traditional benchmarks or processing specifications. Traditional analyses also often ignore the time spent waiting for technical staff to deliver a solution or the delays from several rounds of rework as users react to initial results. A good time-to-result analysis might provide two broad measures:
Time to first query measures elapsed time from the decision to use a technology to when the system can process its first query. It includes contract negotiation, hardware acquisition, software installation, database design, data loading and indexing, front-end tool integration and training for both technical and business staff. Although it may seem odd to include something as nontechnical as contract talks, bear in mind that some cloud-based solutions can literally reduce these from months to minutes. That is a difference worth capturing.
Time to answer measures the time to complete an actual BI project, including problem specification, data preparation, result analysis and subsequent rounds of revisions.
Both of these include conventional measures of labor hours and machine processing time. But they also incorporate the waiting times – for hardware, training, meetings, approvals and fitting tasks into a busy schedule – that are often ignored but determine when a project is really complete.
Like any benchmark, each time-to-result analysis must be tailored to a company’s specific situation. Analysts will need to define scenarios with the particular data sources, staff skills and questions involved. One practical way to develop these scenarios is to build a narrative based on previous projects, starting with the original specifications and tracking how these evolved. This narrative can then be converted into specific tasks, such as adding new sources, refreshing existing data, redesigning queries and creating new reports. Wait times in particular are difficult to assess, because they depend on the priority assigned to any one project. But records of past service levels should provide an estimate of what can be expected on average.
The time-to-result analysis must also distinguish setup tasks from repetitions. Hardware for a new system is only ordered once, while data will probably be refreshed at regular intervals. The theoretical ideal would be to simulate a year’s worth of projects, taking into account the frequency of different tasks as well as the time for each task. In practice, few companies are likely to be so thorough. But they should still keep these distinctions at least informally in mind.
Setup tasks also highlight the difficulty of comparing incumbent technologies with new solutions. The first project with a new technology will incur training costs and delays, while doing the project with an existing technology will not. This is a real difference, so it’s wrong to simply ignore the benefits of using existing resources. But it’s also only a one-time cost, which can legitimately be amortized over several years of projects. Again, this requires envisioning a project portfolio of some sort so you can estimate how many projects to include.
Note that setup costs can sometimes work in favor of a new technology. If the company has steadily expanding or rapidly fluctuating needs, then a technology that makes it easy to add new capacity or train new users will eventually be more economical than an existing technology that is harder to expand. Cloud-based systems, with their low setup costs and brief waiting periods, are the extreme example of how setup costs can provide an advantage over the incumbent.
Of the two measures proposed, time to first query applies more to database engines and time to answer applies more to BI applications. Even though the new analytical database engines are broadly similar in using columnar data structures on shared nothing parallel hardware, there are still substantial differences in how they deal with database design, compression, indexing, incremental updates, new data sources and other tasks. These can impact both the processing and waiting components of time to first query. The impact of these differences also varies greatly depending on the volumes and complexity of the data being processed.
Time to answer is generally more sensitive to the amount of work that end users can perform for themselves. This depends primarily on the interface built into the BI application. Today, the most user-empowering interfaces are attached to the non-SQL analytical databases like QlikView, iLuminate and ADVIZOR. But as BI systems take advantage of the flexibility provided by the SQL-based analytical engines, time to answer will be an increasingly important measure for those databases as well.
In my last column, we started with a simple question: what would BI applications look like if they weren’t designed around the constraints of conventional relational databases?
That’s a lot of ground to cover, but consider the topic: reinvention of BI as we know it. Companies must make many changes to adapt to new technologies, but the results – a huge improvement in the value delivered by BI systems and a huge reduction in cost – are well worth the effort.
David M. Raab is a Principal at Raab Associates Inc., a consultancy specializing in marketing technology and analytics. Raab is also the author of the book The Marketing Performance Measurement Toolkit (Racom Communications). He can be reached at draab@raabassociates.com or via his blog at http://customerexperiencematrix.blogspot.com/.
Be the first to comment on this post using the section below.