Bringing Agility to Business Intelligence
A never-ending list of requests for business intelligence enhancements is the daunting challenge most IT organizations face. It’s no wonder. Forrester Research recently surveyed BI decision-makers and found that 70 percent faced business requirements that changed monthly or even more frequently.
The challenge is only complicated by the fact that traditional BI processes have never seriously been described as rapid, responsive or agile. There is a new way, however, to bring agility to BI processes so that IT gains the upper hand in this battle to address business needs. It requires a new architectural approach.
The rapid change reflected in the number of BI requests is a symptom of our global economy and the new speed of business. Today, businesses are competing not just in efficiency but in their ability to sense market conditions and quickly respond. And in our connected world, market conditions change more quickly than ever before. Whether it’s a global supply chain running leaner through tighter integration with suppliers, or always-on consumers spreading information with smartphones and iPads, any disturbance in the system can ripple throughout the world in minutes. In many cases, this requires decision-makers to have today’s data because last week’s data is just not good enough.
The challenge to the business isn’t just the speed at which information travels or changes, but the diversity of information that has become relevant. Social media, for example, didn’t even exist 10 years ago, and now Twitter feeds, YouTube videos, and community sites have become vital sources of information to marketers, quality engineers and others within the organization. To keep pace, business decision-makers need to collect, share and analyze information from a wide array of sources in a great variety of formats.
BI teams wanting to become more agile, therefore, must not just deliver analytic tools faster and at lower cost. They must also quickly process and present diverse and rapidly changing data sources so that business decision-makers can find the information they need and use the information they find.
Define Agility By Business and Everyday Users
To date, efforts to add agility to BI processes have largely focused on speeding up various steps in the traditional BI delivery cycle, such as the steps to model the data, cleanse the data, build ETL, build the analytic and metadata layers, and create reports and dashboards. This approach has had limited success because the traditional BI architecture is based on a relational data model, which is effective for transactional systems but doesn’t meet the diverse data needs of the business decision-maker.
For example, traditional BI processes demand that data conform to a schema, which can require costly and time-consuming manipulation of structured data. Conforming unstructured content, such as business documents, Web pages and social media to the schema and unifying structured data that was created with different schemas can be cumbersome and often causes IT to exclude data sources altogether. Ultimately, this impairs the user’s ability to interact with the data to find the answers they need.
So while this has helped speed up some processes of reporting on structured data, speeding up existing BI processes has not addressed the need to include diverse and changing data sources. As a result, this approach has not increased the agility of the business – which is the ultimate goal of any agile BI effort. To achieve the desired agility, it’s important, therefore, that agility be defined from the point of view of the business and its everyday users.
Business Leaders Desire Greater Visibility and Ease of Use
Business leaders and their teams articulate some strong desires when it comes to increasing the flexibility of BI systems. In particular, they often demand the BI system provide the following capabilities:
- Visibility to all the data that matters. Business users want IT to unify all the data relevant to particular questions into a single analytic environment so that they can simultaneously view data from varied sources. For example, a marketing executive may need to compare customer sentiment expressed on social media sites with product sales data taken from the company’s order system.
This analytic environment should have the ability to unify structured data from data warehouses, unstructured content (including content from Web sites and social media sites), semistructured content from business documents (including text-based, spreadsheets, pdf, and email documents), data from third-party data sources (e.g., financial reports from Dun and Bradstreet) and information from enterprise systems (e.g., product lifecycle management systems and ecommerce platforms). Business users can answer unanticipated questions and look at complex problems from several angles. In addition, this unified analytic environment can greatly reduce the number of BI requests for IT.
- Easy-to-use analytic interfaces. To improve the responsiveness of their organizations, business leaders want to make information accessible and usable by all those that need it. This requires analytic tools that are designed for nontechnical audiences and approach an online shopping experience where nontechnical users have never needed a training session to determine the best product that suits their multifaceted needs. Systems with consumer-based ease of use ensure adoption, reduce support inquiries, lessen training time and costs, and promote the pervasive use of BI at all levels of the organization.
- Rapid inclusion of new data sources. True agility comes from the ability to respond to changes quickly. Business leaders want the ability to add data sources within days or weeks and/or reflect up-to-date source data, no matter how quickly it changes.
The Emergence of Agile Architecture
There is growing realization that the traditional BI process is too drawn out and simply adding agile development processes doesn’t sufficiently pinpoint business requirements under the new time constraints on the business. There is also realization that the traditional BI architecture does not provide enough flexibility to respond to the needs of the business. Indeed, there is growing awareness that a new approach to BI is needed. For example, Wayne Eckerson, formerly of TDWI, recently wrote: “We need dual BI architectures: one geared to casual users that supports standard, interactive reports and dashboards and lightweight analyses; and another tailored to hardcore business analysts that supports complex queries against large volumes of data.”
Other industry watchers have similarly pointed out that while traditional BI has focused on reporting, OLAP for analytics, or Excel dumps for various information uses, the sweet spot for business decision-makers is to augment a BI infrastructure with a complementary BI architecture, perhaps even a separate “sandbox” containing all the information pertinent to a specific business process that allows decision-makers to explore the data in an unconstrained manner. The sandbox approach doesn’t just have benefits for the business user, though. Because a sandbox can take the place of hundreds of report requests, it can break the cycle of continuous requests for new reports.
This approach addresses a type of decision-making that is not well-met by static reports and not even by parameterized or drill-down reports. The sandbox approach is a better fit for decision processes where the path to the outcome is not clear ahead of time. The questions, therefore, occur in the moment and can’t be anticipated before the analysis is underway.
Just about every knowledge worker faces this type of question dozens of times each day. For example, a warranty analyst addresses questions such as “Why did the number of warranty claims increase last month?” and “Which suppliers can help me reduce my warranty claims?” A marketer faces questions such as, “Why is this distributor generating more profit than the others in the region?” and “Why did that sales promotion work so well last month?” Other workers have similar style questions that pertain to their specific business process. Answering these questions usually requires uninhibited analysis that takes a holistic view of highly varied data sources (not just the structured data from the data warehouse). This type of decision-making process is often called discovery.
To allow nontechnical users to take advantage of a discovery capability, any complementary BI system must address the ease of use requirements expressed by business leaders. Dynamic exploration of the data is usually made possible by:
- Search interfaces. Search capabilities mirror the online search or e-commerce search experience and include faceted search, or an ability to filter by a taxonomy of the data. Search is not just an entry point to allow a user to find a prebuilt report that will then be explored through traditional BI queries, but instead enables all navigation and queries.
- Interactive visualizations. Intuitive visualizations represent data in graphical form and allow the user to explore the data by selecting or highlighting part of the visualization.
Done well, extending existing BI capabilities through a complementary system can provide unexpected discovery capabilities while still maintaining IT’s control and management over the data and the application and, in particular, IT’s existing policies around data governance and security.
Economic Benefits of Agility
With its ability to quickly blend data sources and eliminate BI requests, a complementary sandbox analytic environment can have a sizable impact on IT’s efficiency. IT organizations that have turned to a sandbox approach to replace a traditional data mart development approach have found that they can deploy a sandbox in approximately 20 percent of the time it took to deploy a data mart. The time saved can be used to reduce IT development costs or to expand the number of projects the IT group can address for the business.
With cost as a primary concern for IT organizations, it’s natural for IT to be cautious about implementing any new architectural element. However, the risk of not acting in this case is significant and costly. The visibility and decision-making power afforded by discovery systems can have dramatic effect on the bottom line. These benefits become clear as analytics becomes a primary battleground for business. Many organizations will find they have to retool to handle the diversity of data and adapt to the increased rate of change of business requirements.