Are We Still Business Focused?

Business intelligence should be seen as a business initiative and a business ownership venture. It is - and always should be - about making correct business decisions in not only an improved faster and manner, but also to ensure that it adds value to all business processes. Previously, IT departments operated as part of the company, delivering solutions to the business as soon as possible and with user requirements as the key driver. Today, however, IT departments have started to operate like independent companies within a company, ones that want to make as much money as possible, stretch the work out as long as possible and have business pay for extras that have no immediate benefit to the BI initiative. The truth is that IT departments are taking over the BI initiatives, or in some cases, they are hijacking it. Although this may be a harsh statement, the reality is that data warehouse projects are failing because they are not delivering business requirements on time and within budget and are not end-user focused anymore. 

Some are some articles are available on this subject, and it has always been a problem. Ralph Kimball’s article “TCO Starts with the End User,” has relevant themes: lack of partnership between IT and end users, lack of explicit end-user-focused cognitive and conceptual models, data needed for decisions is delayed and unconformed dimensions. Martin Rennhackkamp, information specialist, Prescient Business Technologies, also highlighted this in an article title “Bridging the great divide between business and IT.” These are all great articles, but in my opinion, the issue has never been highlighted enough or too strongly expressed. 

So where to from here?

Understanding the customer, knowing the customer and believing the customer is king is the modern business mantra. This mantra should always be a part of every BI initiative, data warehouse architectural design/methodology, logical and physical data mart designs and extract, transform and load processes to ensure end-user focus. The disconnect between a project's original intent and its final result is mostly a case of incorrect ownership. As such, the proper project owner, steering committee representatives and project management methodologies should assist with understanding of the original intent by the business - as well as the boundaries the initiative will be developed in.

While IT normally works within the centralized governance guidance, it should be adaptable to the customers’ needs and should have different governance guidance’s for the different business initiatives, (i.e., Application development and BI). As soon as IT starts to overlap governance structures or tries to cross-breed different methodologies, it isn't healthy, irrespective of customers.

When it comes to asking the business, notable problems usually start with generating business requirements. Perhaps IT is phrasing the question "What must this solution do for you?" incorrectly, misunderstanding the answer or perhaps not asking the question at all? Sometimes the right questions are asked and understood, but still need to incorporate other requirements as well.

Wherever IT tries to take sole ownership of a BI initiative, there is a 99 percent chance that it will fail. The main reason for this is where operational issues are brought into or made a part of BI initiatives. IT departments want to clone a transactional system with added history from where the warehouse must be sourced. Departments also want to provide a transactional reporting system without impacting the actual source system. This might be a worthwhile project to implement, but it should not be part of the BI initiative, and the business should not be held for ransom. 

Business has a specific requirement when it comes to BI, and that is a solution for quick and correct answers to specific questions. Data is also growing vastly in volume, and to wait for this “clone” to be loaded before the actual warehouse is loaded can become a major problem with batch window availability.

The following examples are only two of those clones, the first example being a total disaster that actually stopped the BI initiative. The second example is forcing the business to look at other possibilities to assist with their BI needs due to the added time and cost. 

In example 1, illustrated in Figure 1, the data is extracted by identifying the deltas. Only changed rows at the end of the business day are extracted. All source system objects, identified within the source-to-target mappings, were replicated into the Source CDR with additional ETL fields. The data was taken through a series of processes to identify data cleansing issues. Any errors were dumped into buckets along the way, and this would create an unsynchronised target, as the child record could exist without the parent, which also meant that no referential integrity could exist on database management system level. 

The CDR was first called a conformed data repository, then common data repository and then central data repository and lastly, complete disaster repository before it was abandoned. Change request and refresh time take too long, and there is no system of record identification and unreliable data.

In example 2, illustrated in Figure 2, it started off as a great concept. All the necessary architecture, infrastructure and other processes were put in place. Data is extracted by means of delta calculations. Data is taken through different stages to add ETL fields for tracking and history purposes. Surrogate keys are added from staging area to the operational data store area. No data cleansing is physically done, but all errors are reported back to the source systems. 

But, and this is where the problem comes in, whatever the business requirements are for the warehouse, the whole source system is replicated into the ODS in third-normal form. Slowly changing data identification is implemented on each and every table. Even the new surrogate key, for changed records, is stored in the previous record. No referential integrity is enforced on DBMS level. The clone is then used as the only source for any dimensional modeling created further down the line. 

The reason from IT was that this type of design would speed up any changes to the warehouse in the future, supply a complete history of data and make the data available to other business streams as well. The business issue with this design is that it is too costly, implementation takes long and it will push out the data warehouse load time. It will also get more costly over time as every change on the source system, whether it is applicable to the business requirements or not, must be filtered through to the ODS. 

The following is a summary of the issues from a BI and Business user perspective:

Advantages

• Availability of the history for the transactional data systems,
• Data can be cleansed before the warehouse extraction,
• All BI initiatives must source their data from the one single point and
• Additional business requirements in dimensional models can be added with historical data.