Over the last several years, the data warehousing and business intelligence (BI) industry has seen multiple dynamics emerge and subside. From a technology standpoint, it saw a growing number of tool-based vendors emerge, and last year it witnessed significant consolidation. Similarly in the approach toward executing BI projects, the industry began with companies going “big bang” in terms of building their enterprise data warehouse all at once, only to realize later that data warehouse projects are best executed by breaking them into multiple small and manageable phases.


In today’s BI market, one of the buzzwords making the rounds is “frameworks.” The Wikipedia definition of framework is “a basic conceptual structure used to solve a complex problem.”1 As part of this industry, I suppose all of us would agree that BI is a complex problem to solve. So, going by the above definition, the usage of frameworks in this space is absolutely relevant.


A BI framework is a group of reusable components that can be leveraged while developing a BI subsystem. I am consciously using the word subsystem because a framework can be applied to the extract, transform and load (ETL) layers, the underlying data model or even the front-end analytical layer. The most important attribute to understand is that a BI framework can be a concept and does not have to be something tangible or physical.


I will give you an example of a framework that is a concept that does not exist in physical sense but can be leveraged in a BI subsystem design.


In the U.S., the Securities Exchange Commission mandates all asset management companies to file what they call 13F/G/D reports. These are regulatory reports that need to be submitted every quarter and deal with a declaration of the quantity of holdings individual advisors have in specific security classes. (Note: The penalty for noncompliance can usually run into the millions of dollars). From an IT standpoint, rendering these reports involves humongous effort toward collating the data from a wide variety of sources, cleansing it, correcting it and arriving at the correct number. An obvious question is, “Are there any existing frameworks that can help me accomplish this for my business?” If the answer to the previous question is “Yes,” what does that framework look like?


In this particular example, that framework is a concept. The concept is made up of detailed understanding of how an asset management company carries out its business (domain) and a generic data model that supports rendering 13F/G/D reports, which will require customization to make it work in your organization.


To understand the concept you must ask:


  • What does 13F/G/D mean from an SEC standpoint?
  • What are the data elements involved?
  • What are the sources of these data elements from a systems standpoint?
  • What are the known issues you will face when it comes to collating all these data elements into one single report, and how do you solve them?

Imagine yourself trying to execute the project with the answers to these questions versus starting from scratch. Would you have a higher chance of successful execution? The point I am driving at is that in the BI domain, knowledge itself can be viewed as a framework and leveraged.


What Frameworks Can Achieve


The primary driver toward using frameworks is obviously to reduce cost. The unique selling position for using frameworks is that it gives the user a prebuilt platform to start from, and the savings in cost comes from the effort saved using a prebuilt platform.


Not many people relate using frameworks with the quality of the output. This, in my mind, is one of the larger advantages frameworks bring to the table. The fact that a framework exists means that somebody had done a similar project before and the framework encapsulates the knowledge gained from that exercise. The knowledge also comprises mistakes that were made in that exercise and, therefore, will ensure that you don’t commit the same mistakes. Hence, another USP for using frameworks is that it will ensure high quality of the work product that you are trying to deliver.


Now that we have established that frameworks enable reduction of cost and increase in quality, let’s dive into the cost reduction aspect. People who evaluate frameworks typically focus on the development cost reduction, but not too much thought goes into the aspect of maintenance cost.


For example, in the technology stack of a proprietary framework includes a component directed only toward how to handle rejects from your daily load. This rejects handling engine is comprised of a set of database tables, code snippets and methodology that goes a long way in reducing the overall total cost of ownership for the system. When this framework is evaluated, however, the component fails to gather attention simply because the focus is on the cost saving of the development effort.


What a Framework Cannot Achieve


First, a framework is not a product. The expectation that you will plug the framework into your BI landscape and reports will instantaneously be generated is unrealistic. A framework basically helps to jumpstart your BI initiative, but it cannot replace it.


Second, a framework needs to be maintained. If you refer back to the 13F/G/D example, what happens if the regulation itself is changed next year? The framework will need to be tweaked to incorporate changes or else it will become obsolete. Also note that once a BI initiative is complete there will be lessons learned that can be incorporated in your framework to benefit the next project.


Last but not least, a framework is not free. It costs money. It is an encapsulation of knowledge gained while doing similar projects/initiatives in the past. However, while evaluating a framework, you have to look at it in terms of what you are paying upfront and what you will save over a period of time.




  1. Wikipedia. “Framework.” Wikipedia.com, May 15, 2008. http://en.wikipedia.org/wiki/Framework

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access