The challenge is only complicated by the fact that traditional BI processes have never seriously been described as rapid, responsive or agile. There is a new way, however, to bring agility to BI processes so that IT gains the upper hand in this battle to address business needs. It requires a new architectural approach.
The Challenge
The rapid change reflected in the number of BI requests is a symptom of our global economy and the new speed of business. Today, businesses are competing not just in efficiency but in their ability to sense market conditions and quickly respond. And in our connected world, market conditions change more quickly than ever before. Whether it’s a global supply chain running leaner through tighter integration with suppliers, or always-on consumers spreading information with smartphones and iPads, any disturbance in the system can ripple throughout the world in minutes. In many cases, this requires decision-makers to have today’s data because last week’s data is just not good enough.
The challenge to the business isn’t just the speed at which information travels or changes, but the diversity of information that has become relevant. Social media, for example, didn’t even exist 10 years ago, and now Twitter feeds, YouTube videos, and community sites have become vital sources of information to marketers, quality engineers and others within the organization. To keep pace, business decision-makers need to collect, share and analyze information from a wide array of sources in a great variety of formats.
BI teams wanting to become more agile, therefore, must not just deliver analytic tools faster and at lower cost. They must also quickly process and present diverse and rapidly changing data sources so that business decision-makers can find the information they need and use the information they find.
Define Agility By Business and Everyday Users
To date, efforts to add agility to BI processes have largely focused on speeding up various steps in the traditional BI delivery cycle, such as the steps to model the data, cleanse the data, build ETL, build the analytic and metadata layers, and create reports and dashboards. This approach has had limited success because the traditional BI architecture is based on a relational data model, which is effective for transactional systems but doesn’t meet the diverse data needs of the business decision-maker.
For example, traditional BI processes demand that data conform to a schema, which can require costly and time-consuming manipulation of structured data. Conforming unstructured content, such as business documents, Web pages and social media to the schema and unifying structured data that was created with different schemas can be cumbersome and often causes IT to exclude data sources altogether. Ultimately, this impairs the user’s ability to interact with the data to find the answers they need.
So while this has helped speed up some processes of reporting on structured data, speeding up existing BI processes has not addressed the need to include diverse and changing data sources. As a result, this approach has not increased the agility of the business – which is the ultimate goal of any agile BI effort. To achieve the desired agility, it’s important, therefore, that agility be defined from the point of view of the business and its everyday users.
Business Leaders Desire Greater Visibility and Ease of Use
Business leaders and their teams articulate some strong desires when it comes to increasing the flexibility of BI systems. In particular, they often demand the BI system provide the following capabilities:
- Visibility to all the data that matters. Business users want IT to unify all the data relevant to particular questions into a single analytic environment so that they can simultaneously view data from varied sources. For example, a marketing executive may need to compare customer sentiment expressed on social media sites with product sales data taken from the company’s order system.
This analytic environment should have the ability to unify structured data from data warehouses, unstructured content (including content from Web sites and social media sites), semistructured content from business documents (including text-based, spreadsheets, pdf, and email documents), data from third-party data sources (e.g., financial reports from Dun and Bradstreet) and information from enterprise systems (e.g., product lifecycle management systems and ecommerce platforms). Business users can answer unanticipated questions and look at complex problems from several angles. In addition, this unified analytic environment can greatly reduce the number of BI requests for IT. - Easy-to-use analytic interfaces. To improve the responsiveness of their organizations, business leaders want to make information accessible and usable by all those that need it. This requires analytic tools that are designed for nontechnical audiences and approach an online shopping experience where nontechnical users have never needed a training session to determine the best product that suits their multifaceted needs. Systems with consumer-based ease of use ensure adoption, reduce support inquiries, lessen training time and costs, and promote the pervasive use of BI at all levels of the organization.
- Rapid inclusion of new data sources. True agility comes from the ability to respond to changes quickly. Business leaders want the ability to add data sources within days or weeks and/or reflect up-to-date source data, no matter how quickly it changes.
The Emergence of Agile Architecture
There is growing realization that the traditional BI process is too drawn out and simply adding agile development processes doesn’t sufficiently pinpoint business requirements under the new time constraints on the business. There is also realization that the traditional BI architecture does not provide enough flexibility to respond to the needs of the business. Indeed, there is growing awareness that a new approach to BI is needed. For example, Wayne Eckerson, formerly of TDWI, recently wrote: “We need dual BI architectures: one geared to casual users that supports standard, interactive reports and dashboards and lightweight analyses; and another tailored to hardcore business analysts that supports complex queries against large volumes of data.”
