It is considered “hot” to write articles about BI 3.0, with catchphrases like social BI, business analytics and more. And it makes sense to write about these new developments; after all, innovation is necessary to bring the field forward. On top of that, it gives people in the field an opportunity to dream apart from their day-to-day problems.
These day-to-day problems reveal themselves in surveys that show a disappointingly low degree of user satisfaction with respect to BI applications. Too much information is not relevant to the decisions at hand; the quality of information, both the inherent accuracy (of basic data) and the consistency (across sources) is too low; and, above all, it takes way too much time to produce the requested information. So maybe it is more important to do something about these shortcomings instead of aiming for an escape into the future. What can we do about it?
An often-heard reason for dissatisfaction is that the developers of BI applications are not connected with the users. The obvious remedy is: 1) development should take place close to the users and 2) developers should listen carefully to the requirements of the users. Yes, this as a little bit too obvious; and some pundits take this further and state that developers who don’t listen to users should be banned from IT. I admit that obeying this advice will take away some of the pain experienced by the users. However, I’m convinced that the root causes are deeper under the surface and more difficult to address. In my opinion, two issues are ultimately responsible for the majority of all problems with BI.
The first issue, which has nothing to do with BI as such, is the lack of ownership of data, nowadays called lack of data governance. In my opinion, this is the root cause of nearly all problems with data quality. And lack of data quality in transactional data, master data and formulas counts for a major part of the effort necessary to bring data in good order from source systems to BI applications. Data governance is an issue that should be owned and driven by the business, but for a variety of reasons (e.g., politics, complexity, inadequate IT support) it does not get the attention it deserves.
The second issue typical for BI is the pace and degree of change, which is much higher than IT support for operational processes. And IT is, in general, not sufficiently equipped to have an answer for the challenges caused by this level of change. There are two areas where IT is not up to the mark.
- Project and change management approach. Most IT organizations are working with some variant of the waterfall approach. This requires a fixation on the deliverables of the project in a fairly early stage, and that does not align well with the chaotic manner of establishing real requirements of a BI project. The way of working with respect to changes in systems in production is often even more compartmentized, with frequent and linear handovers between different functions. The reason is that IT comes from this background, and today, most applications still support operational processes where this way of working is considered satisfactory, making it difficult to introduce something different for a minority of the applications. But this approach is too static and takes too much time for the hectic world of BI.
- Architecture. The second priority is architecture. BI environments have a history of organic growth driven by users, initially with little or no involvement from IT. The growth traditionally happened at a time when there was little or no attention for architectures in IT anyway. The first “real,” serious involvement of IT took place when the need for an enterprise data warehouse was established. IT people who designed these data warehouses had traditional IT backgrounds, and as a result, the design of these data warehouse was – even in cases where Kimball’s approach was adopted – based on common practices with respect to database design and technologies. Such designs and technologies are not optimal for digesting the changes necessary to keep the logical and technical data model of the data warehouse up to date.
What can be done to achieve a breakthrough in this seemingly deadlocked situation? First of all, I am convinced that the initiative has to come from the IT people. The reason is quite simple: The business has the money, and he who pays the bill decides what music will be played. And as the business doesn’t like the repertoire of the internal IT department, they will bypass them and start shopping outside. This is happening and in a sense represents a move back to the practice of the 1980s and 90s. It can be a valid option for (isolated) applications and will result in short-term satisfaction, but it will lead to long-term problems with consistency and alignment across business units, which are precisely the problems that lead to IT involvement in the first place.
So what can the IT department do to prevent this from happening? Some recommendations:
1. Concentrate all functional and technical expertise and capacity with respect to BI development, change management and even operational support in one organizational unit: the BI Competency Center. Make the BICC responsible for support during the entire lifecycle of the BI portfolio.
2. Adapt a way of working for the front-end applications that is based on prototyping, incremental delivery and evolutionary development. Prototype to the degree where the last prototype is the first operational increment. Allow for a duration of the individual increments somewhere between four and eight weeks. Evolve to the extent that even during the development of an increment, requirements can change on the spot.
When you make these your top priorities, visible results will buy you goodwill and thus time to work on improvements to the architecture that are less visible to users. And a better architecture is probably even more important for long-term, sustainable success.
The puzzle with which you will be confronted in your architecture is how to get data from a variety of operational systems into a variety of BI applications and, in a number of situations, back into the operational systems. All BI applications have their own requirements and characteristics with respect to the business information model, amount of data, number of source systems, complexity of processing, data refreshment frequency, time constraints and number of users.
This is a logistical problem similar to the logistics of physical goods. An application for the analysis of point of sale data in a B2C organization with potentially multiple millions of customers is different from the same application in a B2B organization and may require different technology. An application to analyze the data collected in social networks has yet another profile. On the other hand, applications to support corporate performance management are, as far as it concerns the information logistic structure, quite similar for all industries. Generally speaking, these applications require integration across different systems, aggregation of transactional data and application of formulas. Differences are often caused by the size, complexity, geographical spread and, above all, organizational structure and governance model. This is also the type of application where the first data warehouses were introduced.
The first thing you have to do is to make an inventory of the logistical processes and flows involved. Such a flow consists of transport, decoupling points with or without storage and extraction (intermediate, temporary or long-term), load and other processing stations. Short-term improvements can be achieved by combining the flows with the same characteristics. Long-term improvements must come from the development of an architecture based on an integral vision of the BI environment aligned with the enterprise architecture, if there is one. It is very unlikely that an optimal BI architecture will be based on a single enterprise data warehouse where all data is collected and from which all BI applications are refreshed. It also very unlikely that there will be an architecture without any data warehouse at all.
Thus, the most likely outcome will be an architecture with a number of data warehouses and some BI applications directly on top of a single operational system, from which the data model and the history are good enough to have an application without any decoupling point. Different data warehouses might be using different database technologies, especially in situations where very large numbers of consumer transactions are involved. Independent of your specific situation, it is very important to implement a kind of federated model where data warehouses share the same dimensional structures.
It also imperative that the business information model of the organization - and not the logical data model of the individual data warehouses - is the starting point for the design. The business information model and thus the logical data models in your data warehouses/data marts can and will change frequently and suddenly. Because this takes a lot of time, I recommend using technology that allows you to generate the data warehouses and data marts.
And yes, preparing your architecture also involves looking to the future and what can be expected as far as developments in your user organization and the technological trends you have to take into account. Once a data governance program is in place, your information logistical process will be simpler and less complex to maintain. So there is nothing wrong in being the advocate for such a data governance initiative, but you should’t be the owner of it.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access