Business intelligence should be seen as a business initiative and a business ownership venture. It is - and always should be - about making correct business decisions in not only an improved faster and manner, but also to ensure that it adds value to all business processes. Previously, IT departments operated as part of the company, delivering solutions to the business as soon as possible and with user requirements as the key driver. Today, however, IT departments have started to operate like independent companies within a company, ones that want to make as much money as possible, stretch the work out as long as possible and have business pay for extras that have no immediate benefit to the BI initiative. The truth is that IT departments are taking over the BI initiatives, or in some cases, they are hijacking it. Although this may be a harsh statement, the reality is that data warehouse projects are failing because they are not delivering business requirements on time and within budget and are not end-user focused anymore.
Some are some articles are available on this subject, and it has always been a problem. Ralph Kimballs article TCO Starts with the End User, has relevant themes: lack of partnership between IT and end users, lack of explicit end-user-focused cognitive and conceptual models, data needed for decisions is delayed and unconformed dimensions. Martin Rennhackkamp, information specialist, Prescient Business Technologies, also highlighted this in an article title Bridging the great divide between business and IT. These are all great articles, but in my opinion, the issue has never been highlighted enough or too strongly expressed.
So where to from here?
Whos the Boss?
Understanding the customer, knowing the customer and believing the customer is king is the modern business mantra. This mantra should always be a part of every BI initiative, data warehouse architectural design/methodology, logical and physical data mart designs and extract, transform and load processes to ensure end-user focus. The disconnect between a project's original intent and its final result is mostly a case of incorrect ownership. As such, the proper project owner, steering committee representatives and project management methodologies should assist with understanding of the original intent by the business - as well as the boundaries the initiative will be developed in.
While IT normally works within the centralized governance guidance, it should be adaptable to the customers needs and should have different governance guidances for the different business initiatives, (i.e., Application development and BI). As soon as IT starts to overlap governance structures or tries to cross-breed different methodologies, it isn't healthy, irrespective of customers.
When it comes to asking the business, notable problems usually start with generating business requirements. Perhaps IT is phrasing the question "What must this solution do for you?" incorrectly, misunderstanding the answer or perhaps not asking the question at all? Sometimes the right questions are asked and understood, but still need to incorporate other requirements as well.
IT versus Business
Wherever IT tries to take sole ownership of a BI initiative, there is a 99 percent chance that it will fail. The main reason for this is where operational issues are brought into or made a part of BI initiatives. IT departments want to clone a transactional system with added history from where the warehouse must be sourced. Departments also want to provide a transactional reporting system without impacting the actual source system. This might be a worthwhile project to implement, but it should not be part of the BI initiative, and the business should not be held for ransom.
Business has a specific requirement when it comes to BI, and that is a solution for quick and correct answers to specific questions. Data is also growing vastly in volume, and to wait for this clone to be loaded before the actual warehouse is loaded can become a major problem with batch window availability.
The following examples are only two of those clones, the first example being a total disaster that actually stopped the BI initiative. The second example is forcing the business to look at other possibilities to assist with their BI needs due to the added time and cost.
In example 1, illustrated in Figure 1, the data is extracted by identifying the deltas. Only changed rows at the end of the business day are extracted. All source system objects, identified within the source-to-target mappings, were replicated into the Source CDR with additional ETL fields. The data was taken through a series of processes to identify data cleansing issues. Any errors were dumped into buckets along the way, and this would create an unsynchronised target, as the child record could exist without the parent, which also meant that no referential integrity could exist on database management system level.
The CDR was first called a conformed data repository, then common data repository and then central data repository and lastly, complete disaster repository before it was abandoned. Change request and refresh time take too long, and there is no system of record identification and unreliable data.
In example 2, illustrated in Figure 2, it started off as a great concept. All the necessary architecture, infrastructure and other processes were put in place. Data is extracted by means of delta calculations. Data is taken through different stages to add ETL fields for tracking and history purposes. Surrogate keys are added from staging area to the operational data store area. No data cleansing is physically done, but all errors are reported back to the source systems.
But, and this is where the problem comes in, whatever the business requirements are for the warehouse, the whole source system is replicated into the ODS in third-normal form. Slowly changing data identification is implemented on each and every table. Even the new surrogate key, for changed records, is stored in the previous record. No referential integrity is enforced on DBMS level. The clone is then used as the only source for any dimensional modeling created further down the line.
The reason from IT was that this type of design would speed up any changes to the warehouse in the future, supply a complete history of data and make the data available to other business streams as well. The business issue with this design is that it is too costly, implementation takes long and it will push out the data warehouse load time. It will also get more costly over time as every change on the source system, whether it is applicable to the business requirements or not, must be filtered through to the ODS.
The following is a summary of the issues from a BI and Business user perspective:
- Availability of the history for the transactional data systems,
- Data can be cleansed before the warehouse extraction,
- All BI initiatives must source their data from the one single point and
- Additional business requirements in dimensional models can be added with historical data.
- No consolidation between systems takes place,
- No linking between common business areas,
- Unnecessary data and processing,
- History data is already stored within the warehouse,
- Building every table in the ODS as a slowly changing table is very resource intensive and slow,
- No validation and integrity checking,
- Loose parent-child relationships as parent can arrive later than the child due to quality issues and error handling,
- Data quality is moved from source to clone, creating versions of data, if data quality is not pushed back to original source,
- Changes to source objects, whether its used within the warehouse or not, must be implemented,
- No business specification is taken into consideration,
- Very high cost before actual data mart is developed and
- No quick ROI.
After reading both examples, it seems less to state that there is a failure in BI initiatives to deliver the business requirements and they are not end-user focused anymore.
With the increasing use of consulting companies to assist IT departments with the implementation of a BI initiative, success of such an initiative can be achieved, but it is important to make use of a company with a proven track record. Some companies are still under the impression that a BI project is a simple summarization of relevant source data. Many application-focused companies think they can build a data mart because it has significantly fewer tables and processes than a transactional system and is only used for reporting. This unfortunately is where the whole BI initiative goes down the drain. Businesses lose money, and the BI communitys reputation takes a big knock.
Solutions are available, and a lot of success stories prove that business and IT can get back on track to make the BI adventure a joint initiative. It has taken years to prove to business how important BI is to the successful running of a company. Lets not go back to the dark and unintelligent ages. Lets play nicely. Treat business and IT initiatives as a parent-child relationship - listen and understand each other. With this in mind, great successes can be achieved and BI can grow.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access