Testing 1, 2, 3
Rob Farris would like to thank Mahesh Pethe, CBIP, senior manager at Hitachi Consulting for contributing this months column.
BI projects, like most IT projects, are managed using a variety of software development lifecycles. The user acceptance testing activities are typically the last set before the project is signed off on and deployed. It is the final point in the project lifecycle to increase and ensure quality before it truly matters - once the product is deployed defects truly impact the business. UAT is typically the first time that real users gain an understanding of the product, view and verify the data, use its full functionality and provide valuable feedback. Therefore, UAT is a critical part of any BI project lifecycle.
While there are many case studies, tools and templates about the overall testing activities (in particular for the unit, system, integration and performance testing subphases) there has not been enough attention paid to UAT, especially for BI projects. Sometimes, UAT planning does not occur until just before UAT is scheduled to start, due to project timeline pressures and/or other issues. As a result, the UAT phase is often not planned or executed well.
What is the impact of inadequate, poorly conducted UAT? For BI projects, where information is provided to senior leaders and executives and used to make strategic business decisions, poorly conducted UAT can be disastrous, including:
- Poor or suboptimal business decisions,
- Frustrated users,
- Dissatisfied customers and sponsors
- Increased cost,
- Reduced profitability,
- Reduced ability to change the organization due to a lack of trust in the data,
- Lower-quality product,
- Large amounts of time spent fixing issues in production and
- Opportunity cost of building the next release or a new product.
Here are some guidelines to properly plan and execute UAT on typical BI projects in order to avoid some of these pitfalls.
There are many challenges during planning and execution of UAT. It is important to acknowledge and address these in order to end up with a high-quality product that has the strong backing of the project sponsor.
- Lack of strong sponsorship. The sponsor/business owner must help UAT testers understand the importance of the product and be willing to spend the time needed to resolve any issues. Without this commitment, it will be difficult to get the commitment from the testers.
- Availability and willingness of testers. Many testers have other full-time responsibilities. A strong business owner will help gain testers commitment to perform thorough testing. The testers may have to reprioritize their regular responsibilities to accommodate UAT, which may require approval from their leadership. However, it is critical that the UAT testers are committed and engaged during UAT.
- Availability of a UAT platform with the right data. Ideally, UAT should be conducted in a separate testing environment. However, there can be issues with acquiring such an environment and loading all the data typically necessary for a BI project. Budgetary, licensing and data security issues can prevent or restrict the provision of a pristine environment. The risks associated with not having such an environment must be clearly documented to the customer and a n alternate plan should be devised for how UAT will have to be conducted using a shared environment.
- Connectivity to the test platform. All UAT testers and the project team will obviously need access to the UAT environment. This can be a challenge for global users, users connecting through virtual private networks and users connecting through firewalls from other corporations. These kinks need to be worked out before UAT starts. Otherwise, valuable test time will be taken in getting the testers connected.
- Setting up UAT testers correctly. This can be a significant challenge. In many BI projects, different user roles have to be set up, each with different access and security requirements. As testers are set up with these roles, problems can be uncovered - this can impact security requirements, resulting in delayed UAT. Therefore, it is important that all security roles and profiles be tested before UAT is scheduled to commence, or adequate time must be allowed to account for these setup issues.
- Availability of the proper tools. It is important to select an issue-tracking tool during UAT so that all testers can log and sign off on issues. If UAT testers cannot access this tool, alternatives must be developed so they are engaged in the UAT process.
- BI projects, in particular, present data challenges. Do UAT testers understand the data, metrics and source systems? Do they understand the aggregations, slices, trends, signals, dials, and charts that are part of the new product? Pre-UAT training may be required in order to bring the testers up to speed on these new data elements in order to mitigate these challenges.
- BI projects are often met with resistance from end users. New insights into how the business is running, while critical for leadership, can often put employees on the defensive. Perhaps they are worried about their jobs. If they are responsible for producing the reports, will the new product make their jobs obsolete? BI projects can also be part of the leadership toolkit that is trying to affect change in the business. UAT testers (and eventually end users), could resist that change, which could manifest in unnecessary complaints about the tools, noncritical issues or even not signing off on the UAT phase. Thus the UAT phase itself may need to be run as a mini change-management project.
- Clearly defined project acceptance criteria that are signed off on by the customer. These are critical to the creation of a good test plan. Acceptance criteria provide guidance during UAT to determine if fixing a defect is in or out of scope. This is critical for BI projects in particular, because many defects found during UAT of BI projects can be attributed to issues in the source systems.
Communication, Interaction and Language Differences:
- Time zone differences. For a global solution, the team of UAT testers may be located around the globe. This presents logistics challenges, including how to communicate and interact seamlessly across global time zones. These types of issues need to be worked out before the start of UAT.
- Testers prefer various communication models. Will your UAT group prefer meetings, conference calls, email and/or instant messaging? If their expectations are not set during the kickoff of the UAT period, issues can be overlooked and opportunities lost for a higher quality product can be missed.
- Ability to clearly understand one another with minimal ambiguity. Global UAT testers from non-English-speaking or even from English-speaking countries can experience communication challenges. There can be translation challenges and ambiguity in wording for example. Identification and resolution of any issues can be extended by days due to miscommunication or ambiguity. Extra time must be allocated to account for these situations.
In summary, BI projects face both common and unique challenges during their UAT phase. The key to successful UAT is to conduct adequate planning in advance of its start.
My next column will provide specific recommendations and tips addressing these challenges and how to maximize success of UAT on BI projects.