Testing 2, 3, 4
Rob Farris would like to thank Mahesh Pethe, CBIP, Senior Manager at Hitachi Consulting for contributing this months column.
Business Intelligence projects are managed using a variety of software development life cycles. User acceptance testing activities are a critical part of any BI project lifecycle, because they are usually the final set of testing activities before the project is signed off and deployed into production.
Part 1, I discussed the many challenges faced during the UAT phase of BI projects. Part 2 examines how to address and overcome those challenges as well as how to maximize the success of UAT.
In order to mitigate the many potential UAT challenges, the UAT phase must be planned well in advance of its commencement. For a typical four- to six-month BI project, UAT planning should start during the design/build phases, or a minimum of two to four weeks before its actual start.
Prior to commencing the UAT phase, complete these critical milestones:
- Clearly define and sign-off on project acceptance criteria.
- Successfully complete all unit, system and integration testing of every BI component, including database schema, database scripts, extract, transform and load packages, online analytical processing cubes and reports.
- Create UAT test cases and test scripts.
- Develop a UAT environment that includes the right data. Decisions about the test data - such as how much data, breadth of the data and whether the data needs to be secured, masked, or treated in any way - must be made and confirmed with the sponsor before UAT commences.
The following are recommendations for the UAT planning tasks:
- Strong project sponsorship is required to make UAT successful. The sponsor or business owner needs to identify UAT testers and obtain their commitment. It is recommended that the sponsor, rather than the project team, identify testers and communicate with them directly so that they understand the importance of the project and their role.
- UAT testers must be a representative sample of the target user community. You will need testers representing each applicable region, industry, product, customer, organizational unit, country, etc. Testers representing all user roles are required, such as sales, project, administrator and executive.
- The number of UAT testers depends on the size and complexity of the application. Some testers will be more active and provide more feedback than others. The sponsor should consider the culture of the user community in order to make this judgment. It is recommended that between six and 15 testers be identified for a typical four to six month BI project.
- The UAT timeline must be planned and communicated to the testers well in advance of the actual start date, so they can adjust their schedules and accommodate the tasks.
- There should be at least one week in the project plan allocated to setting up committed testers with their roles, security levels and access.
- The UAT timeline needs to be adequate to address all potential issues, but still be realistic. Too little time results in low-quality product. Too much time results in tester fatigue, minor nonvalue-add enhancements and delayed launch. For a typical four to six month BI project with six to 15 testers, a three- to four-week UAT testing period is usually appropriate.
UAT Execution For proper execution consider the following reccomendations:
- A UAT kickoff meeting should be scheduled with mandatory participation for all UAT participants a week prior to the start of user acceptance testing.
- Day one of UAT is critical. It sets the stage for a successful product launch. During the UAT kickoff meeting, the project team must convey that this phase of the project is highly collaborative and that input and participation is required in order to launch successfully.
- It is recommended that you use an online tool to log defects, issues and questions.
- You should plan for the team to conduct a short (30 minute) daily UAT touch point meeting where all the participants are encouraged to attend.
- The goal of the touch point meeting is to triage newly logged issues, bugs and questions.
- Discuss these to determine if they are in fact bugs to be fixed. If not, decide what actions, if any, need to be taken. Any confirmed bugs should be assigned to the appropriate party on the project team. Remaining questions should be assigned to the project sponsor or business owner. The goal of this meeting is not to resolve issues, but merely agree on how to proceed. Therefore, it is important to keep the meeting moving
During the meeting, go through status updates on open items, and confirm items that are fixed and can be closed. The frequency of this meeting can be reduced as UAT progresses and the number of issues decreases. The project manager can make a judgment call on the frequency needed. For a project with testers located globally, have two daily meetings at opposite ends of the day, so that testers from all time zones can participate.
Project managers should use an incentive-based approach to encourage testers to be engaged. For example, the project managers can reward testers with small prizes on a daily or weekly basis. Best bug of the day/week or best data issue of the day/week prizes can be awarded to increase participation and have some fun during what can sometimes be a tense phase of the project. This can increase the collaborative spirit of the team and eventually lead to a higher quality product.
Issue Tracking Tool
There may already be a standard tool used to track incidents or issues; if so, great. Use a Web-based tool that allows for multiple concurrent user access rather than a spreadsheet on a file-share. If no specific incident tracking tool is available, then a Web-based portal such as Microsoft SharePoint, if available, can be used for this purpose.
All UAT participants must have access and be set up on whatever tool is selected prior to start of UAT. If workflow functionality is available, it should be leveraged to effectively assign various issues to different owners, throughout the lifecycle of the issue.
The UAT timeline should consist of three inter-related activities:
- Having testers execute test cases and logging issues,
- Getting the BI team to respond to these issues and make the agreed-upon fixes and
- Allowing testers to validate that the changes have been made and sign off on them.
Although these activities can happen concurrently, the percentage of time spent on each of them varies with time, as shown in the Figure 1. The focus of activity should change depending on which stage of UAT is occurring
Communication and Interaction Among Teams
UAT testers may prefer a different form of communication, such as email, conference calls, instant messaging or Web meetings. The UAT team must work actively with testers to ensure that communication is taking place effectively between groups. It is the responsibility of the project team to work with the testers using their preferred approach whenever possible.
The BI team must push the use of the issue log tool as the best means of tracking communication, yet adapt to users needs for the various mechanisms. For example, in cases where an issue begins as an email thread, it may be a good idea to copy the thread into the tracking tool, so it can be assigned and addressed effectively. Sometimes, issues cannot be completely analyzed or discussed in the daily half-hour checkpoint call. In such cases, a more detailed meeting or call should be set up with the specific testers and any other required participants to further discuss and resolve.
The BI team should work aggressively to meet the UAT timeline and resolve outstanding issues. The recommended approach for closing user acceptance testing is for the sponsor to send out a vote email with a deadline, that allows testers to provide a final yes, no or yes with reservations vote. Based on the responses, the sponsor should provide the final go or no go decision that is, give the business approval to launch the product or decide to postpone the launch. If there is a no go decision, the project team should re-do the project timeline and eventually re-execute the UAT phase to obtain sign-off.
These guidelines should help overcome and avoid the many challenges associated with UAT. With proper planning and communication, you can work together to maximize UAT success and help increase the quality of your companys BI projects.