Vitally critical information can be literally "hidden in plain sight." This fact is cleverly illustrated in Edgar Allan Poe's classic detective story The Purloined Letter, in which a corrupt government official has hidden a stolen document in his apartment. The police can't find the letter despite tearing the place apart. Poe's detective Dupin shocks the police by easily finding the letter, hidden in plain sight on a desk alongside other innocuous papers. Dupin realized, just as the sly government official had, that the police would look for the letter only in places where they themselves might hide it and would overlook the letter sitting right under their noses!


The critical task of providing actionable enterprise data often falls victim to the same limited thinking displayed by the police in Poe's story, particularly when such mistaken logic is applied to laudable but misguided efforts to implement pervasive BI across an organization. Many IT professionals mistakenly believe the only way to get enterprise BI into the hands of those who need it is to implement BI tools, typically with still-complicated interfaces, as well as create and maintain the data management systems necessary to support BI.


Fortunately, there is a much more pragmatic approach to achieving the aim of pervasive BI: getting the right information to the right people at the right time. This article will introduce report mining, an enterprise solution that delivers on the promise of pervasive BI and is based on an understanding of how people really work, not how IT professionals think people should work.


Pitfalls of Conventional Pervasive BI Solutions


Some would argue that pervasive BI represents the wave of the future where high-level executives are fully supported in their business decision-making by all members in an organization. By connecting to the appropriate database with sophisticated data visualization tools, information is available to anyone looking to analyze it. But the question remains, does everyone need to analyze this information? In reality, pervasive BI can turn out to be more trouble and extra work with no incremental company benefit.


Giving all levels of users this type of universal access is an inefficient approach because of the multitude of challenges associated with using data warehouses or transactional databases as the source for pervasive BI. These challenges range from the scalability and security issues of potentially hundreds, if not thousands, of users banging against such data sources to the likely end-user missteps that may impact performance. Organizations also need to consider the extensive IT support necessary to train end users in the use of BI tools. Additionally, a database-centric approach may hamper a company’s ability to share analyzable information with people outside the immediate enterprise, such as employees in distributed offices, customers, vendors or partners.


While allowing end users to access information by their own accord is acceptable in some instances, there frequently are other situations where the need for BI does not warrant the time nor the effort required to extract information out of a transactional database or data warehouse.


Microsoft Excel: The True Pervasive BI Tool


There are, at best, a few million copies of BI tools licensed worldwide, yet more than a third of these seats are shelfware.1 In contrast, there are more than 200 million licensed copies of Microsoft Excel in active, constant use on a daily basis.


Excel is the epitome of pervasive BI, providing what virtually all knowledge workers really want: information in a format they can immediately understand and use with confidence. For years, BI vendors have been swimming against the current by offering BI tools that enterprises "should" use instead of Excel; solutions intended to be superior or sufficiently “Excel-like” to replace Excel in the organization. However, these efforts are often in vain. As Forrester Research recently observed, "spreadsheets are a permanent fixture in enterprises because no other analytical application outperforms them in flexibility, ease of use and ubiquity."2


Of course, the simplicity and flexibility of Excel is also a significant disadvantage for BI professionals trying to ensure the sanctity of the "single version of the truth" in an organization and avoid the malady of multiple business users deriving multiple results from their respective spreadsheets. Indeed, Forrester observes the need for "tools that will bridge the gap between spreadsheet management and spreadsheet usage in the BI process."3


Forrester goes on to review a variety of solutions intended to provide enhanced spreadsheet management and quality control of spreadsheet data. This review includes the possible use of enterprise content management (ECM) solutions to archive and manage spreadsheets centrally. Forrester correctly asserts that traditional ECM solutions used for the purpose of storing existing spreadsheets do not provide any value-add to the BI process.


However, a different and new ECM-related approach will bring a sudden windfall of value-add to the BI process. Instead of the mere storage of existing spreadsheets, this pragmatic approach builds an innovative BI bridge between ECM and the existing reports already produced by the organization's enterprise information systems. Many IT and business professionals still mistakenly disregard existing reports as static, read-only documents. As a result, existing reports often remain hidden in plain sight as a programming-free source of pervasive BI.


Report Mining: Merging the ECM and BI Worlds into Enterprise Information Management (EIM)


Enterprise information systems typically offer a vast library of existing or “canned” reports and business documents (invoices, purchase orders, bills of materials, etc.). Such systems, either managed in-house or by an outside data processing firm, range from legacy mainframe systems to top-tier enterprise resource planning (ERP), HR/payroll and industry-specific core processing systems for banking, health care, retail and other industries. For example, SAP offers canned reports numbering in the several thousands.


What many executives don't realize is that the existing reports already produced within their organization are their greatest assets because most knowledge workers already use reports for decision-making purposes. Therefore, putting those reports to use should be a top priority. The problem is that conventional wisdom still says reports cannot be used as a source of BI because they are static; they do not allow the end user to ask questions about the data and see different data views.


Thanks to report mining, this is no longer true: existing reports can now easily become live, actionable data for easy analysis, with no new database programming required. Report mining is a technology that enables, on demand, the intelligent recognition and parsing of data within existing reports, typically in plain text or PDF format, into a valid data table. The new data tables are complete with calculated fields of data and database lookups to include additional data located elsewhere. With this technology, users can sort and filter data as well as combine data tables, summarize data into subtotals and grand totals and easily export data to Excel.


As a result, reports - intelligently indexed and archived within a report mining solution - suddenly become a wellspring of easily accessible and customizable data for programming-free, Web-enabled pervasive BI, with the reliable and consistent on-demand creation of standardized Excel spreadsheets. By unlocking important data that already exists in reports, organizations can provide users with actionable information quickly - in the way they need it.


In contrast, traditional BI systems rely on "structured" information (databases), which typically comprise only 20 percent of all enterprise information. Report mining, on the other hand, unlocks a vast portion of unstructured and semi-structured data sources, notably existing reports and business documents, as a new Web-enabled source of BI. Plus, reports contain business rules, and in many organizations, especially financial, any information generated out of databases has to be checked against audited, verified reports for accuracy anyway. So, why not use the original reports as the source of pervasive BI?


Report mining technology is ushering in a new holistic approach to enterprise BI, more accurately described as enterprise information management (EIM). EIM strives to deliver pervasive BI using all existing enterprise information sources, particularly existing reports, which still remain hidden in plain sight in far too many organizations and are typically ignored by traditional BI systems.


Unique Report Mining Benefits


One of the drawbacks of typical BI systems is that they cannot integrate information provided to the organization in the form of a report or business document. Examples of these types of documents include reports provided by outside data processing firms for organizations that have outsourced their core transaction processing, or special industry-specific documents provided by an outside entity, such as explanation of benefits (EOBs) and other health care-related documents. Report Mining solves this problem through its ability to turn any kind of report into actionable information, thus eliminating third-party customization requests and reducing the risk of errors while saving time and money.


The noninvasive nature of report mining can also readily translate to reduced total cost of ownership (TCO) of an organization’s enterprise information systems. Because users of a Report Mining solution are accessing data mined from archived reports instead of, for example, live data from a production ERP system, the organization can significantly reduce its number of live seats to the ERP system. Live seats can be reserved for those workers who actually process transactions and otherwise regularly interact with the system, eliminating licenses for users who simply connect to the ERP system for the express purpose of data queries and running reports.


Finally, and perhaps most intriguingly, a report mining solution will yield data which is not possible to obtain using any BI tool. Many enterprise information system reports include complex calculations, such as derived inventory valuations, specialized financial calculations and more. These calculations do not "exist" as data within the database; they are automatically performed at report runtime. A BI tool would have to replicate these same calculations from scratch and confirm the data’s accuracy by comparing the BI tool's results with the original report. Therefore, the spectrum of available enterprise data actually expands through the use of report mining.


Just as Dupin amazed the police in Poe's story with a seemingly effortless retrieval of the stolen letter after their own fruitless exhaustive efforts, so too can a Report Mining solution astound enterprise knowledge workers with its dramatically faster and easier implementation cycle, extreme ease-of-use, and ability to deliver on the promise of pervasive BI: the right information to the right people at the right time. Report mining solutions also fully leverage spreadsheets while simultaneously ensuring the spreadsheets are populated with consistent data and customizations, all at a fraction of the complexity and cost of traditional BI solutions. There is no better time than now to take a long, hard look at existing reports as a surprisingly simple, powerful and “hidden in plain sight” source of actionable enterprise data.




  1. Neil Raden. Business Intelligence 2.0: Simple, More Accessible, Inevitable. Intelligent Enterprise, February 1, 2007.
  2. Boris Evelson. “Ouch! Get Ready – Spreadsheets Are Here to Stay for Business Intelligence.” Forester Research, April 2, 2007.
  3. Boris Evelson. “Ouch! Get Ready – Spreadsheets Are Here to Stay for Business Intelligence.” Forester Research, April 2, 2007.


Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access