This is the first in a series of "Problem Solved" columns written by various consultants of Baseline Consulting. Each column will address a real-life project or client challenge and discuss how the author addressed the issue.

A colleague who was aware that I'd been managing a major client initiative using offshore resources recently asked me how my project was going. The project had initially been met with skepticism but actually delivered very successful outcomes.

After hearing of our success, my colleague related the details of his latest project that was delivered late and cost significantly more than expected. These experiences are familiar to anyone who's worked a large data warehouse.

IT departments often assume that outsourced projects are less complex than in-house projects. Management and people issues render outsourced projects more complex, requiring a higher degree of  oversight. Data warehouses, which are frequently viewed as strategic, up the ante in terms of the need for rigor and expertise. Simple processes for manageability, teamwork and productivity were created that helped us avoid common data warehouse outsourcing pitfalls.

The most prevalent pitfall in outsourcing a data warehouse project is a misunderstanding of specifications at the onset of the project. In BI projects, the prevailing development method is iterative, characterized by short projects that deliver enough functionality to provide value quickly and engage end users. This can strain even a colocated team, but when developers are in different locations, the risks such as code rewrite, bugs and missed deadlines are even greater.

Productivity issues - either real or perceived - loom large with outsourcing. A project manager who normally interacts with the staff in person can become overwhelmed by needing to communicate with offshore teams. Even with the lure of discounted hourly rates, any project interruption can idle dozens of programmers. Project interruptions are a greater risk when developers are thousands of miles and 12 time zones away.

Time zone differences (and limited overlapping work hours) mean delay. Scheduled events can often be pushed into the next day or even week. As my team begins our Friday workday, the day in India is already over.

Without significant experience with the offshore partner firm, skill levels can be a concern. Some offshore firms are renowned for training their junior staff members on the client's nickel. As I'll discuss, the assumption of zero control over offshore staff skill levels is far from the truth.

In any data warehouse or BI project, monitoring progress is a challenge, especially when the majority of the development team is on another continent. After all, you can't just walk over to someone's cubicle to ask what milestones are being met or missed each week.

The key to avoiding these pitfalls is recognizing the details that make an offshore project more complex to control than an in-house project. In our case, we carved out specific development tasks and created simple processes that were easy for everyone to follow.

My team used a hybrid development approach that combined elements of iterative and waterfall delivery methods. Using this approach, we deconstructed the data warehouse development process into phases, carefully defining discrete roles within each phase. Using our existing standards and templates, we defined the deliverables that drove the initiation of each phase. For example, a detailed design specification and sample data file were delivered to the offshore team to initiate the coding phase. This hybrid approach allowed us to:

  • Manage development by monitoring the phase deliverables;
  • Scale the process because we understood the phase dependencies, skill level and workload requirements for each phase;
  • Accurately estimate the amount of work required to build extract, transform and load (ETL) programs; and
  • Be flexible and "light" enough to reach two- to three-month delivery targets.

In fact, we designed our development organization to optimize offshore productivity. We staffed on-site lead engineers from the outsourcer to design the ETL code and to manage a dedicated offshore ETL programming team. The lead engineer bridged the time zone gap by collaborating with onsite staff and management and communicating to the offshore team. Senior-level client participation and support were critical to this success.
We vigilantly enforced minimum skills and experience requirements. We reviewed resumes and interviewed candidates. Building a great team improved our overall productivity and significantly reduced the number of bugs reported.

Managing a complex data warehouse implementation with standard project management tools can be onerous. We simplified project monitoring by tracking the deliverables that were dependencies for each development phase. We tracked the availability of every phase input; for instance, design and test specifications and outputs such as ETL code. The tracking documents were maintained by the offshore developers and used by the project manager to develop management status reports.

How do you avoid these pitfalls in future outsourced data warehousing projects? Simplify your development processes to reduce the complexity. Some rules of thumb to keep in mind:

  • Design your organization to simplify the communication between teams.
  • Insist on experienced offshore developers for your project and hire senior engineers on site who have experience managing offshore teams.
  • Use a hybrid development methodology that takes the best parts of iterative and waterfall methods and formalize the requirements specification processes.
  • Simplify managing offshore activities by tracking the development phase deliverables, inputs (specifications) and outputs (deliverables).

Outsourcing data warehouse development will always be a challenge. My advice is to remember the premise behind Occam's Razor: the simplest solution is most likely the best. Problem solved!

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access