This column is focused on measuring business intelligence (BI) success. Topics will cover The BI audit (TBIA) BI Capability Maturity Model, which is a blueprint for the audit of BI assets.
In this fourth and last column in the series of articles on measurement factors for BI assessment (i.e. the key performance indicators), the focus will be on the six KPIs highlighted in Figure 1. All the KPIs have been selected for the BI capability maturity model from business and technical drivers that are well recognized within the industry as keys to success. These are mostly common sense, straightforward, and just a repeat of well known industry terminology. These KPIs are important, however, in the manner in which each will be used in the BI audit as measurement factors.
The next set of articles will describe the BI asset base, all components and key assessment features.
Does the information get to the right user at the right time? In most cases, performance measurements in the BI arena mean response times, i.e., the time from the request for information to the return of an answer to the user. This is fine as long as there is an agreed upon and clear definition of the terms. Service level agreements (SLAs) are an important key to ensure consistent Business Information Quality. In any such service level agreement between the BI end user and the IT group, it is important for both groups understand and agree about the specifics. The right time for the user may be within five seconds, two days or two weeks. The IT specialist should know what is possible and only agree to a reasonable right time. These SLA response times should be correlated to the types of information queries, because there may be different response times associated with simple queries compared to complex reporting.
Ease of Use
A BI user may be interested only in standard reports. The BI user may also be a power user or analyst whose expectations are for complex, analytical capabilities. BI products must meet the requirements for each user, in a manner which is intuitive for that user. The interface should also evolve naturally with the capabilities of the user. We are moving toward more structured BI analytical programs and processing, which are focused on carefully leading the user through simple to complex research/processing. Check out the capabilities of your organization during the audit.
Do we have all the information that is required? During the audit, for every part of the BI asset base, this is a key question for assessment. This can apply to a review of the analytical platform and reporting referring to the answering of a single question, a BI application or a complex analytical process. This assessment should be made, as well, for the Data Integration Platform and that single source of truth, the information presentation library. Have we anticipated the data needs for the organization to meet all the business objectives for some planned period of time?
How accurate is the information? Is it accepted as truth across the organization? How many replications of the same data element exist in how many separate locations across the organization? How do we keep them in synchronization? Or, do we even try? Some of the source data systems for the data warehouse (DW) may be using the same data fields in multiple ways? Do we check that out before we collect and dump it into a DW? Is there a gold or certified copy of the data somewhere in the organization which we know can be accepted as accurate? Are there stewards for the data who are responsible for maintaining the quality of the data? Is the information accurate, timely, up to date and is it consistent across the organization? Does anyone or everyone trust it? Product quality at the heart is data quality. Defining data quality is relatively easy. Ensuring data quality is extremely difficult.
How do we audit for product quality in a BI audit? We ask all these questions, and more, and assess the data. This means searching out a data quality initiative that may have been started three years ago and then disappeared. It will most often mean asking the users for anecdotal evidence of problems or good experiences. Problems that have occurred should be easy to find and relatively easy to track. The audit specifics depend on the organization, the business asset base component being assessed and what the research uncovers within the organization. Very often, it is not pretty.
The term, velocity, in this context describes how fast data moves from entry into the system through the interim processing and becomes information available for use. The business requirements are changing and we are moving quite rapidly, as an industry, into the realm of real time or near real time expectations for information availability. This KPI becomes more critical as the requirements and expectations within the organization evolve. There are some limited best practice assumptions that can be made for how long the data should take from point of entry into a DW and through to the end user.
However, how fast we pick up the data from the source systems is another question. Do we need to have a federated, or similar, technology to map and access directly from the operational, real time systems with a later entry into some data integration platform, such as the DW? Or, can we wait until that data can be processed and integrated into a DW i.e., a few hours or overnight? The organization must decide how fast they need the data based business requirements. The BI audit must include some identification of velocity goals. A BI audit needs to assess what is happening now within the organization. Do goals exist and are they being met? The audit team also needs to be aware that vendors are offering federated systems directly to business users. These systems can then extract the data and place it into a convenient database for that specific business user. This is a new avenue for proliferation of silos of data. One of the most useful results of a BI audit is to provide an understanding of exactly what is happening related to the data across the organization. In most cases, IT cannot stop the process. However, with recognition comes, at least, the ability to face the issues and try to resolve them.
Value to Cost
How much did the BI cost in time, money, and human resources? How much value did and are we receiving from it? We use this KPI as an overall ranking factor for the organization. This is probably the most difficult of our audit measurements. A reasonable figure can usually be calculated for costs for the overall organization using total resources expended and other costs. Estimates of Value returned are always more difficult and less easily accepted. On another level, we expect (or hope) that any IT project which makes changes or adds to the BI Assets will include some value to cost (cost/benefit) estimates. These are measurement factors which can be important to management in prioritizing and planning BI efforts for the organization. The crucial requirements here for a BI audit are the following goals:
- Obtain and get some management agreement for value to cost estimates for the organization. This will assist in the ranking of the BI capabilities of the organization.
- Identify and assess any practices being used for BI projects.
- Determine how the organization is using the application and infrastructure enhancement project estimates.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access