These day-to-day problems reveal themselves in surveys that show a disappointingly low degree of user satisfaction with respect to BI applications. Too much information is not relevant to the decisions at hand; the quality of information, both the inherent accuracy (of basic data) and the consistency (across sources) is too low; and, above all, it takes way too much time to produce the requested information. So maybe it is more important to do something about these shortcomings instead of aiming for an escape into the future. What can we do about it?
An often-heard reason for dissatisfaction is that the developers of BI applications are not connected with the users. The obvious remedy is: 1) development should take place close to the users and 2) developers should listen carefully to the requirements of the users. Yes, this as a little bit too obvious; and some pundits take this further and state that developers who don’t listen to users should be banned from IT. I admit that obeying this advice will take away some of the pain experienced by the users. However, I’m convinced that the root causes are deeper under the surface and more difficult to address. In my opinion, two issues are ultimately responsible for the majority of all problems with BI.
The first issue, which has nothing to do with BI as such, is the lack of ownership of data, nowadays called lack of data governance. In my opinion, this is the root cause of nearly all problems with data quality. And lack of data quality in transactional data, master data and formulas counts for a major part of the effort necessary to bring data in good order from source systems to BI applications. Data governance is an issue that should be owned and driven by the business, but for a variety of reasons (e.g., politics, complexity, inadequate IT support) it does not get the attention it deserves.
The second issue typical for BI is the pace and degree of change, which is much higher than IT support for operational processes. And IT is, in general, not sufficiently equipped to have an answer for the challenges caused by this level of change. There are two areas where IT is not up to the mark.
- Project and change management approach. Most IT organizations are working with some variant of the waterfall approach. This requires a fixation on the deliverables of the project in a fairly early stage, and that does not align well with the chaotic manner of establishing real requirements of a BI project. The way of working with respect to changes in systems in production is often even more compartmentized, with frequent and linear handovers between different functions. The reason is that IT comes from this background, and today, most applications still support operational processes where this way of working is considered satisfactory, making it difficult to introduce something different for a minority of the applications. But this approach is too static and takes too much time for the hectic world of BI.
- Architecture. The second priority is architecture. BI environments have a history of organic growth driven by users, initially with little or no involvement from IT. The growth traditionally happened at a time when there was little or no attention for architectures in IT anyway. The first “real,” serious involvement of IT took place when the need for an enterprise data warehouse was established. IT people who designed these data warehouses had traditional IT backgrounds, and as a result, the design of these data warehouse was – even in cases where Kimball’s approach was adopted – based on common practices with respect to database design and technologies. Such designs and technologies are not optimal for digesting the changes necessary to keep the logical and technical data model of the data warehouse up to date.
Remedies
What can be done to achieve a breakthrough in this seemingly deadlocked situation? First of all, I am convinced that the initiative has to come from the IT people. The reason is quite simple: The business has the money, and he who pays the bill decides what music will be played. And as the business doesn’t like the repertoire of the internal IT department, they will bypass them and start shopping outside. This is happening and in a sense represents a move back to the practice of the 1980s and 90s. It can be a valid option for (isolated) applications and will result in short-term satisfaction, but it will lead to long-term problems with consistency and alignment across business units, which are precisely the problems that lead to IT involvement in the first place.
So what can the IT department do to prevent this from happening? Some recommendations:
1. Concentrate all functional and technical expertise and capacity with respect to BI development, change management and even operational support in one organizational unit: the BI Competency Center. Make the BICC responsible for support during the entire lifecycle of the BI portfolio.
2. Adapt a way of working for the front-end applications that is based on prototyping, incremental delivery and evolutionary development. Prototype to the degree where the last prototype is the first operational increment. Allow for a duration of the individual increments somewhere between four and eight weeks. Evolve to the extent that even during the development of an increment, requirements can change on the spot.
When you make these your top priorities, visible results will buy you goodwill and thus time to work on improvements to the architecture that are less visible to users. And a better architecture is probably even more important for long-term, sustainable success.
