Based on telephone interviews with 100 companies, this article reports on an in-depth study of the critical changes impacting the business intelligence (BI) marketplace. The publication sponsor for the study is DM Review, whose subscriber base of 72,000 professionals served as the contact list. To qualify for the study, business and IT managers were asked whether they are directly involved with existing BI systems or are planning a significant investment in BI systems within twelve months.
The driving force of changes in the BI market is e-commerce with its digital economy of instant global transactions that require integration across legacy systems. Companies are rethinking and retooling their business processes as part of moving toward e-commerce. The need for actionable intelligence about customers, suppliers, distributors, partners and competitors becomes increasingly critical. As such, BI has become a pivotal component for any effective e-commerce initiative. BI enables many of the cutting-edge developments in customer relationship management, supply chain analysis, sales force automation, distribution channels and technology forecasting.
Most agree that the general objective of BI is to make businesses perform smarter and, therefore, better. However, there is little agreement about the details behind that objective.
From a historical perspective, the BI marketplace has traditionally been focused on management decision support utilizing the technologies of data warehousing, query/reporting, multidimensional analysis, data mining and, more recently, information portals.
This set of technology has served us well over the past decade. BI has reduced risk and decision time in dealing with unconventional, unanticipated and messy business situations. Because of this success, the BI marketplace is under stress, forcefully driven in diverging directions by smart horizontal applications such as customer relationship management, innovative vertical solutions such as those in telecommunications and healthcare, and advanced technologies such as knowledge management and text mining.
Some think that BI systems will be fragmented into little pieces and embedded in point applications, never to be seen by IT eyes again. Others are often unclear as to how the dynamics for BI systems will unfold over the coming years. Hence, there is a huge uncertainty within IT groups for planning and implementing BI systems and within vendors for developing BI products and services.
Our findings confirm several traditional notions of BI and some suspected trends. However, the useful findings are the many surprises that run counter to industry wisdom.
BI systems are no longer confined to internal functions. They are impacting many areas external to the enterprise.
Respondents confirmed the traditional purposes and functions for BI systems, such as periodic management reporting. In particular, according to 75 percent of the respondents, "ad hoc analysis as business problems arise" is the top purpose for BI.
However, unexpected were the external impacts of BI systems. A significant number of companies are "supplying intelligence back to our customers" (53 percent) and reporting "to suppliers and partners" (19 percent). Users of BI systems go beyond internal management with support for customers (43 percent), business partners (35 percent), suppliers (21 percent) and distributors (17 percent). Access to the warehouse environment was widening to "Web-based dissemination to all users" (36 percent) and "personalization of information to all users" (34 percent). Marketing (79 percent) and customer relationships (70 percent) are running a very close second to traditional BI applications such as financial analysis (82 percent) and profitability analysis (70 percent).
BI systems are no longer limited to an elite group of executives but, instead, are touching the entire hierarchy. In the past, BI systems catered to a small number of executives and their staff to analyze financial issues. Although executives (75 percent) and line management (72 percent) are the top users, clerical personnel are considered BI users in 52 percent of the companies. Forty-five percent of the companies now support more than one hundred users, with 25 percent supporting 750 people or more.
IT is a major gatekeeper for BI systems but many parties (including customers) are involved. When asked who has responsibility for BI within the enterprise, 73 percent indicated that a single group has that responsibility. Nearly half of those (41 percent) indicated that this group is the IT unit responsible for the data warehouse (DW); another 40 percent indicated that this group is a committee of both IT and business unit people. Over all companies surveyed, the IT unit responsible for DW has overall BI responsibility about 30 percent of the time.
When asked about purchasing authority, 89 percent indicated that upper IT management signed off on BI purchases, with roughly half indicating a spread across several groups. Further, the champions for BI initiatives are also spread across multiple groups with no clear pattern. It is surprising, however, that BI initiatives often involve customers (40 percent) and business partners (22 percent).
Data warehousing is the cornerstone of BI systems. Eighty- seven percent of the companies have one or more warehouses in operation, of which more than half have several. In fact, four companies have twenty or more warehouses which, one can only hope, are logically integrated in some fashion. Eighteen percent of these warehouses are a terabyte or more in size. Thirty percent expect their warehouses to double or more over the next two to three years.
The diversity of data sources that feed the warehouses is surprising. Every warehouse uses transactional systems as a data source. However, several unusual sources were mentioned, such as e-commerce Web sites (51 percent), linkages to databases located at customers or partners (49 percent), data enhancement services that add customer characterization information (47 percent), Web site extractions (41 percent), search/discovery services (34 percent) and news data feeds from content aggregators (33 percent).
Finally, 47 percent of the companies have one or more data marts, and 49 percent are using an operational data store.
Figure 1: BI System Purposes and Functions
Privacy in BI systems is in an immature state. Twenty-two percent of the companies collected personal information, such as address and phone numbers. When asked about privacy policies, only 11 companies (out of 22) have a comprehensive policy in place, and four companies have little or none. It may be prophetic that only one person noted that privacy legislation would be major factor.
Security is a concern, but adequate security has yet to be realized. Twenty-four percent of the companies indicated that they have an "extremely high" level of security in BI systems; however, 45 percent indicated that they desired such a level. In other words, there is a significant gap between actual and desired levels of security within BI systems.
The internal IT staff is where the action is, and they expect instant implementations that quickly generate business value. Respondents indicate that internal IT staff is involved in BI implementations (70 percent) and that IT involvement will increase (66 percent). In contrast, only 23 percent indicated that system integrators (SIs) are involved in BI implementations, with only 33 percent indicating that SI involvement will increase. A surprising result is that a small 8 percent are using service providers (SPs) to supply a specific BI service; however, 33 percent expected that SPs would increase their involvement.
When asked about the appropriate ROI horizon for BI investments, respondents (66 percent of those doing ROI justifications) indicated that BI projects should have a payback of less than one year.
Figure 2: BI Applications
The BI marketplace is healthy and growing. In the past, BI was a small segment within other markets such as database management and reporting tools. Now, the BI market is driving rapid growth in leading-edge applications such as CRM. Most respondents (89 percent) expect growth in the BI market, with 23 percent expecting the market to "explode" in the future. Everyone perceived some business value for BI, with 50 percent feeling that BI delivered "very high value."
Thirty percent said their budget for BI systems was $1 million or more. Six percent of the companies have a BI budget of $50 million or more. There is no obvious correlation between BI budget and industry. Their BI budgets have a healthy growth rate with 25 percent of the companies expecting 30 percent or more and eight percent expecting a doubling or more over two to three years.
BI vendors should emphasize integration and then make their products inexpensive, easy to develop and easy to use. When respondents were asked about their perception of the "next BI wave," we collected a wide range of responses. Many related to "integration" as a key theme. Respondents felt that the next BI wave would be the integration of BI among various facets across the enterprise. An analogy is the dynamic that ERP has had on the operational side of enterprise systems. A similar dynamic is desired or expected on the informational side of enterprise systems. Other BI waves dealt with CRM, Web enablement and portals.
Figure 3: BI Users
In another question, respondents were asked to rate various BI waves as to importance. The scalable data warehouse was the top BI wave with 63 percent rating it as "critical." Other BI waves of importance are: business performance indicators (57 percent as critical), CRM (46 percent), enterprise information portals (43 percent), real-time closed-looped data analysis (33 percent) and external information sources (31 percent).
When asked about their concerns and suggestions for improving BI, respondents hammered consistently on the same themes: lack of proper skills, too expensive, too difficult to use and too difficult to develop.
Based on the results of our survey, here is some advice for BI practitioners.
- Explore ways of packaging and disseminating BI information to customers, suppliers and other key business partners. How can this information make their lives easier and build loyalty to your company?
- Explore how external data sources can enhance and extend warehouse functions.
- Understand and document privacy issues that are impacting your company and recommend the adoption of industry best practices.
- Improve the skill levels of the internal IT staff in BI concepts and practice.
- Explore the delivery of BI functionality through the use of BI service providers.
- Consider the overall enterprise architecture with respect to BI systems, particularly in the management of meta data and business processes.
- Audit the ability of your system architecture to scale gracefully. Successful BI systems often cause surprising increases in demands.
Likewise, BI vendors are advised as follows:
- Make products easier to develop and install, well within the abilities and capacity of the typical IT staff. The IT staff should be confident that they can take your product and rapidly generate benefit to the company.
- Make your products easier to use, with most common functions well within the ability of middle and lower levels of the company. Do not just "dumb-down" the product by eliminating functions. Achieve a simple but effective design whose interface is naturally understandable to the person in a specific job position.
- Investigate delivering your product as a service through some variation of the service provider (SP) model. Design your product so that it can be massively configurable and instrumented into the SP infrastructure. This is a small segment currently but is perceived as having high growth potential.
- Frame your value proposition so that you can honestly and clearly argue that your product has a payback period of less than one year.
- Nurture relations with the IT group responsible for the data warehouse. Understand who champions BI initiatives and who is involved in decisions for implementing BI systems.
- If you have a point solution (regardless of its marvelous benefits), balance your emphasis with a healthy amount of enterprise integration. Argue and demonstrate that your solution will play and integrate well with other solutions and lead to a coherent enterprise architecture.
- Engineer your products and services so that they are extensible and scalable. They should serve as a starting point for smooth functional evolution toward future solutions and provide a solid foundation for rapid growth.
- Design your products to accommodate flexible privacy policies and insure robust security measures.
The Role of BI Today
In conclusion, this study paints an image of a pervasive and comprehensive role for BI in today's global enterprise. BI is no longer your mother's warehouse. We have evolved into a much richer, broader and comprehensive notion of business intelligence, inconceivable for most only a few years ago.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access