The experiences of the early adopters of business intelligence (BI) and data warehousing provide important lessons about what works in business intelligence solutions. These lessons and emerging trends in BI technologies reveal the principles that will lead to greater value from your business intelligence investment.
Is "business intelligence" really an essential business tool or an oxymoron? The answer will depend upon the level of success your company has had with both business intelligence and data warehousing.
The first 10 years of BI and data warehousing (DW) initiatives have resulted in many successful, high-return applications of information technology. Looking back on the first decade of BI and DW, this article will present these critical lessons, identify the principles behind them and point the way to successful enterprise-level BI.
The lessons learned are based on experiences in a variety of companies that have employed BI technologies. They are independent of technology but highlight the principles that, if followed, can create successful BI results. However, while much can be learned from the early adopters of BI technology, it seems that many companies and organizations are not paying attention to these hard-earned lessons and are in danger of repeating mistakes we know how to avoid. Not knowing the history of BI and DW may result in repeating doomed initiatives. In fact, real-world experiences have helped us identify four critical BI best practices, trends in BI technologies and applications, and management factors that affect strategic BI success.
Four Critical BI Best Practices
As data warehouses and data marts have been built and the BI infrastructure has evolved, several key best practices have surfaced that clarify the approaches needed for BI/DW technical success. These best practices focus on BI/DW architecture, legacy system data cleanup, BI data organization and intelligence presentation.
BI/DW Architecture: The architecture of the BI/DW process is the fundamental area that affects the long-term success of the BI/DW solution. The problem occurs when there is a focus on data marts rather than on a single enterprise data warehouse. This creates a series of data marts filled with data culled from locations that initially seemed appropriate, but for which there is not a long-term strategy for maintaining the data or enterprise meta data.
Over time, the dependencies along this ad hoc flow of data become unreliable because of changes in the applications, data stores, data marts or data warehouses located upstream. Essentially, this (lack of) architecture is a higher-level form of the chaotic flow of data at the application-system level. The first BI principle is to create stability in the basic structures of data fundamental for providing BI and running the business.
The best practice solution is to develop a hub-and-spoke architecture. This architecture has a central data warehouse that feeds the data marts (see Figure 1). While changes in applications and their data stores will certainly continue, this architecture ensures that only the extract, transform and load (ETL) process that extracts and transforms the data from the source application is affected. All data marts supplied by the data warehouse are isolated from these changes.
Figure 1: Hub-and- Spoke Architecture
Legacy System Data Cleanup: Another key area is focusing the data warehouse on the quality, integrity and usability of data. It is easy to take data from an existing application system and simply copy it into a data mart or data warehouse. This approach is often used to build a data mart or data warehouse quickly. If data is not cleaned before loading, the data must be embedded in the analytic queries that create BI reporting. Data cleanup, in its simplest terms, includes determining for each data element its standard physical characteristics; its source system of record; and its decoded, most basic structure. Every step must be taken to make all data in the warehouse equivalent in meaning, as opposed to value, to its enterprise system of record.
The second BI principle is that data used for BI must ensure that each data element stands on its own as a fact or attribute. Data that requires processing in order to become usable is likely to create misleading results because it is difficult to ensure that the rules for decoding the data are always properly employed. This creates the problem of keeping all report queries consistent and correct in applying decoding rules; otherwise, conflicting and inconsistent BI results would be produced.
The best practice is to embed required decoding rules in the data ETL process (see Figure 2). The end objective is to have the data in the data warehouse stand on its own. Any inconsistencies that appear in subsequent BI results will then be the result of a poorly constructed query rather than the quality of data in the data warehouse.
Figure 2: Apply Business Rules as Part of ETL
BI Data Organization: Data organization may seem straightforward. However, experience in mergers or regional realignments has demonstrated that much too often the organization structure is used as the data organizing principle for creating data marts. This may satisfy current operational needs but limits the ease with which enterprise-wide BI can be developed because the same kind of data (customer billing data, for example) is distributed into multiple data marts.
As any employee knows, organizations change all too frequently, as do the application systems that support them. These changes require modifying one or more data marts whose data is structured around the organization. The BI principle is to keep an enterprise-wide focus for BI and supporting data marts or warehouses, not to focus on departmental, regional or other category functions.
The best practice solution is clear: Develop a single data mart or warehouse that contains all required enterprise and local (departmental, regional, etc.) data elements. This approach makes the data mart useful for local purposes and supports analysis of data at the enterprise level as well (see Figure 3). While there will still be changes in the organization of the enterprise, this approach will not alter the number of data marts and will simplify responding to organizational changes. Typically, such a change will only add or change values to a data or dimensional fact contained in the data warehouse.
Intelligence Presentation: The presentation of BI is often taken for granted. The focus is on providing the data analytics required to support the business and then delivering all the resulting BI to the business community for which the system is designed. This situation occurs when BI technology is used as a state-of-the-art report generator. This overwhelms business managers who need BI to help them make informed decisions without reviewing volumes of analyses and query results. Overwhelming business people with information creates a common problem, and BI gets lost like a needle in a haystack.
Figure 3: Develop a Single Target
Ultimately, BI must not be viewed as simply the analytical report. The BI principle to apply is to present the information a manager or executive needs in order to make an informed decision. Data presentation must go beyond providing analytic results and include tailoring the results to the decision-making needs of the businessperson.
Another best practice is to use the "80:20" rule providing the businessperson the 20 percent of the analytic results that provides 80 percent of the business impact. BI uses thresholds to separate critical results from noncritical ones. Present data using graphics and other tools to help managers and executives understand the BI. All of these techniques need to be tailored for the individual receiving the BI results in fact, all individuals should be able to choose their own methods of separating critical from noncritical results. All BI results will be available, complete with drill-down capability, to the business manager or executive as required; however, immediate attention should be on critical BI (see Figure 4).
Figure 4: Threshold- Based Exception Notification
Trends in BI Technologies and Applications
The ongoing technological change in the IT industry also affects BI. These changes come from three sources: the incursion of the Internet into all aspects of the enterprise, BI innovations and an emerging business trend of demanding more than online analytical processing (OLAP) queries and reporting. It would take a much longer article to provide a thorough technology review, but a few general observations can be made.
The Internet: It is a maxim that the Internet is changing the way businesses operate. For BI, the changes are in the form of using Web browsers to access BI results and to initiate ad hoc queries. The Internet is also allowing intelligence to be shared across corporate boundaries so that companies and their suppliers can share essential information almost moment to moment. Shared intelligence and browser access has caused BI vendors, as one would expect, to make their products browser-based. Browser-based access has become the dominant method for presenting BI to business people.
Browser-based access to BI has created a competitive overlap between technologies. BI vendors are adding portal-like capabilities to their products, and portal vendors are adding BI-like features to theirs. It appears that portals cannot really provide BI capabilities, nor can BI vendors provide full enterprise portal capabilities. BI vendors will need to compete based upon the BI capabilities and innovations their products offer.
BI Innovations: The Internet itself has opened up a new source of BI competitive intelligence. Competitive intelligence uses crawler technology to search targeted references on the Web. These references can be about competitors and their products, the presence on the Web of your company's name and products, and other desirable intelligence such as market data. This is clearly a desirable direction for BI because it can provide decision-makers with timely competitive intelligence.
There are other innovations as well: intelligence presentation and visualization technologies are emerging to transform volumes of complex data into comprehendible, rather than overwhelming, intelligence. Pharmaceutical companies analyze data that describes the structures of their existing drugs and chemical formulations to find matches that may apply to conditions caused by genetic structures. This approach is used to shorten the time it takes to identify promising new medicines. Similarly, petrochemical, processing and manufacturing companies are analyzing their operational variables to find optimal process and manufacturing conditions. This will lower operating costs and increase product quality. In cases such as these, the large number of variables and the complex interrelationships between them make it difficult to grasp the significance of the results. Intelligent presentation technologies can help decision-makers in every business as they analyze their own complex, interrelated operational variables.
Pattern analysis is another information technology innovation that has been applied very successfully in certain areas. For example, in high- volume, real-time areas such as credit card transactions, telephone calls and financial trading, pattern analysis has been used to detect potential fraud. Using mathematical techniques such as heuristic analysis, neural networks and fuzzy logic, pattern analysis will begin to be applied to BI as a means of helping business people process the increasing volumes of BI that come their way in the world of e-business.
Predictive analysis is also gaining momentum as an essential BI tool. This technology requires a data warehouse with a large volume of historic data that can be analyzed for correlations. These correlations may be useful for predicting information such as the likely market response to a new product, the best test markets to evaluate a product introduction, operational bottlenecks and their causes, and so forth. While no prediction is certain, predictive analysis provides insights that are not easily available from other technologies.
Emerging Business Trends: The technology innovations described in this article are not yet widely used, nor is the market picture clear about the viability of the products or vendors in all BI tool categories. What is clear is that BI is becoming much more than OLAP data analytics. Where many organizations still use OLAP data analytics as a report generator, a growing number of companies are applying a variety of BI technologies for competitive and strategic advantages.
As companies transform their businesses through e-commerce, customer self-service, integrated supply chains and other advances that promote interactions with customers and suppliers, electronic business operations can only be understood and managed through BI technology. The world of e-business is the world of advanced, strategic BI. Clickstream analysis and Web analytics are of little use unless they are integrated with customer and operational data.
Companies are recognizing the need to integrate Web-generated data with business data and recognizing that no single BI technology can provide all the tools required for success. Thus, another principle is that strategic BI must use several different BI technologies that "play well" together (see Figure 5). The only way in which this can occur successfully is with an integrative platform for BI that provides a single source of meta data to manage the consistent and correct application of a variety of BI tools.
Figure 5: BI Technology
Management Factors That Affect Strategic BI Success
The first generation of BI concentrated on the operational monitoring requirements of business units. These requirements focused on what business managers and staff needed to know in order to run their business operations. Typically, "need to know" questions were specified in terms of the data to be analyzed and how the BI process should present it. The business managers and staff from the business unit provided these specifications and often became comfortable using the technology (usually OLAP) for their own ad hoc queries.
Strategic BI addresses questions that management does not even know to ask. Examples of these questions are: How can I enhance revenues? What new business opportunities are available? What losses are being incurred and how can they be prevented? What cost containment opportunities are available? Strategic BI provides information that helps decision-makers answer these questions.
Businesses are generally quite good at running their operations and solving operational problems. To do this effectively, they are organized managerially and financially around business units; however, strategic BI investigates questions for which no single business unit, manager or executive is responsible. This presents a significant challenge for the enterprise to get the greatest benefits from investment in BI.
Reviewing the principles behind BI best practices and technology innovations reveals some interesting points about managing BI strategically. The principles are to:
- Create stability in the basic structures of data fundamental for providing BI and running the business.
- Ensure that each data element stands on its own as a fact or attribute.
- Keep an enterprise-wide focus, not a departmental, regional or other category focus.
- Make BI not simply the analytical report, but the information a manager or executive needs in order to make an informed decision.
- Use several different BI technologies that "play well" together.
Stability and clarity in data are classic BI principles. However, it is the last three principles that truly point the way to successful BI.
A single, corporate center for BI can safeguard and apply these principles for the enterprise. The center for BI, in my view, should report to the CIO who ensures that BI delivers value to the enterprise. BI uses corporate data, requires specialized technologies used by trained professionals, serves the entire enterprise and must meet the needs of all BI users with a consistent level of service excellence and information. Successfully achieving this objective requires knowledge and skill in BI architecture, strategic BI focus (similar to a results and ROI orientation) and managing BI as a core IT and business process for corporate success.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access