Project managers know a critical success factor in any project is good requirements gathering. For business intelligence initiatives, this is particularly crucial. Because data is at the heart of business intelligence, failure to clearly understand data requirements will almost always increase project cost and time. It could also cause the project itself to fail.

To address this issue, a two-step requirements gathering process, discovery and deep dive, will yield higher quality requirements and reduce project risk.

Step 1: Discovery

The discovery step is the first iteration of requirements gathering. The results of this step are expanded and enhanced in the subsequent deep dive step. The discovery step has five main activities.

Technical review: The development team needs to understand the technical landscape in which it will be operating. This includes:

 

  • Hardware used in the environment,
  • Operational and developmental software used,
  • System interfaces and
  • Existing project processes – standards, approvals and documentation artifacts.

A thorough understanding of these elements will allow the development team to focus on project-specific tasks. Ensuring development team members are granted all appropriate permissions to system and physical environments is a simple but essential task to be completed immediately. All too often, a project will fall behind schedule in the early going simply because routine accessibility issues were not resolved at the outset.
Top level reporting requirements:  Reporting requirements fall into three categories: recreate existing reports, create new reports and create an ad hoc reporting capability. It is not unusual for a project to have requirements in all three categories. While an organization may have a main goal of reproducing existing reports from a data warehouse, new reports and ad hoc reporting capabilities are frequently needed as well. 


For every required report or ad hoc query, it is essential that a report owner or subject matter expert be identified.  This person will be accountable for providing the detailed requirements about the information needed. If the report owner is not able to directly provide the requirements, he or she is responsible for finding resources within the organization that can. 

Interview report owners: For existing reports, the following points must be documented:  

 

  • Report purpose,
  • Report recipients,
  • Data element definitions,
  • Calculations and aggregations and
  • Source of data  and update frequency.

 
If a new report is needed, the report owner needs to provide the same information. Often, the best way to accomplish this is to create a report mockup which is a realistic facsimile of the desired report.  If ad hoc reporting is required, then the report owner needs to create realistic, comprehensive and specific query scenarios that will be analyzed just like a report.  

All reports and scenarios developed in this step are repurposed as test cases to be used in future steps. Careful and thorough analysis is essential so test cases truly reflect the intended requirements of the project. 

Document and validate reporting requirements: Based on interview results, all target reports are “dissected.” That is, reports are analyzed column by column, row by row. Individual report elements are analyzed to trace source data back to the tables from which they originate. Calculations are documented to the lowest level of detail. 

If additional information is called for on a new report, the report owner should give specific feedback on what additional data elements are needed and what calculations and aggregations need to be performed.

Once all the reports and scenarios have been documented, a review by the report owner will validate completeness and accuracy.

Perform preliminary analysis of source data: Once the report analysis has been validated, then the source data tables are tested from two perspectives.

Data Quality. What is the condition of the data? Are there nulls in columns that were otherwise thought to the 100% populated? Are there values in columns that are unexpected (e.g. – alpha characters where only numerals were expected)?  

Data Relationships. From the sources that have been identified, can data be joined as specified in the report analysis?   For example, a query scenario may call for reporting shipments by customer. The shipments master would be tested to see if it can be joined to the customer master. It is not unusual for seemingly simple data relationships to turn out to be much more complex in the source data than even the most knowledgeable report owners are aware.

The results of the data relationships analysis are a major reality check for the project. If data quality and/or data relationships problems are more significant than expected, the scope of the project might be affected. If this is the case, the project manager needs to meet with the appropriate stakeholders to reset expectations and amend the project scope as necessary.

Step 2: Deep Dive

While the objective of the discovery step is to gather initial requirements, the deep dive expands and enhances these findings with the objective of making sure data in the project is thoroughly understood. Despite the considerable amount of work undertaken in the discovery step, it is a mistake to avoid the deep dive. This step results in critical enhancements to requirements already uncovered and frequently reveals important new requirements.

In the deep dive, several of the activities of the prior step are repeated. Report owners, SMEs and other stakeholders are visited again to review the discovery documentation in detail. Clarifications and amplifications by these individuals are encouraged in order to correct and refine the requirements.

At this point in the project, meetings with larger groups of stakeholders may be necessary. Sometimes interviews with one (or more) SME are not adequate. Problem-solving meetings with a larger group may be the most effective way to answer tough, complex questions that have been raised in one-on-one discussions. Such meetings also serve to create consensus and keep the larger constituency within the organization aware of the progress and direction of the project.

Evolving Toward Extract, Transform and Load. It is especially important that the appropriate stakeholders once again review in detail the report analysis. This analysis documents each data element specified in existing or proposed reports and queries, tracing them back to the tables from which they originate. Final validation of the report analysis is a clear indication that stakeholders agree that the requirements for the project have been properly captured. Although report requirements and ad hoc query requirements were thought to be finalized in the discovery step, it is common for stakeholders to add requirements that were previously overlooked.

The deep dive then takes the information from the report analysis and begins to develop the first draft of the source-to-target (STT) document. In the ETL process to be designed for the project, source data is extracted, transformed as necessary and loaded into the target tables in the data warehouse. The process of evolving the report analysis into a STT document focuses more specifically on how the source system data will be transformed (cleansing, calculations, aggregations). While the STT will be refined in the process of creating the ultimate business intelligence design, a first pass at what will be the ETL roadmap is an important part of the final requirements. 

Scope changes. Just as the discovery Step may uncover the need for a scope change, new information revealed in the deep dive may be of such magnitude that scope must be re-assessed at this step as well. 

In these cases, a change order must be prepared and approved. Such changes can have a negative impact on the timeline if they are not handled effectively. There needs to be a mutually agreed upon change management process, quick identification and documentation of scope changes, and swift decision-making on whether or not to approve a change.

Determining cardinality. An additional document produced in the deep dive step is the cardinality diagram. This document spells out the one-to-one, one-to-many or many-to-many relationships in the data. Capturing knowledge of these relationships is central to understanding the underlying data. The cardinality diagram needs to be validated by the appropriate SMEs since erroneous assumptions about cardinality will profoundly affect BI design activities.

Preliminary star schema. As a precursor to the design, a preliminary star schema is developed. Under the heading of “floating a trial balloon,” a preliminary star schema gives the development team an idea of what the possible architecture might be for the final deliverable. This preliminary design is then used to create the final deep dive deliverable, prototypes.

Creating prototypes. In the discovery step, sample source data was tested for quality and to determine if basic data joins could be accomplished as expected. In order to finalize requirements, simple report prototypes are built. The objective of the prototypes is to further test the data to determine if the requirements to be handed over to the development team truly reflect the needs of the project.  Clients are especially interested in seeing prototypes because they not only provide a more user-friendly way to visualize requirements, but they also create a concrete sense that the project is progressing.

If the set of reports to be delivered is large, then a smaller set of reports that cover the widest variety of data is selected for prototyping. Also, the prototyping tool does not need to be the same reporting platform as will be used for the final deliverable. All that is required is a rough representation of how the final product might look.

Employing a two-step process to gather requirements may seem like overkill, especially when project timelines are tight. Stakeholders may complain that the additional time and expense are not worth it. However, experience has shown that if requirements are not sufficiently detailed and clear, a project has a much higher risk of exceeding its time and cost budget. Making the effort up front to execute both the discovery and deep dive steps will greatly enhance the project’s prospects for success.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access