Should the Data Warehouse be Outsourced?
One of the top issues for CIOs is the outsourcing of IT. Most CIOs are being asked by their CFOs to strongly consider outsourcing and offshoring many of their IT functions, the prime motivation being cost savings - more specifically, short-term cost savings. The first challenge is to determine what to outsource and offshore and what to keep in house. What about the data warehouse?
In an outsourced situation, contracts are very clear about what will and will not be included. The very nature of a data warehouse is that the users are never able to articulate all their requirements initially, meaning each new major opportunity requires renegotiations and contract changes. By the time these details are worked out, the opportunity might be lost.
Let's look at each of the major activities in the data warehouse and evaluate which of these activities are appropriate for outsourcings.
Data Modeling. While there are modeling commonalities within a vertical industry, every organization has its own way of doing business; these unique processes should be included in the models. An outside consultant might know the industry very well, but it is doubtful he or she would be aware of the organization's unique processes. Data modeling should not be outsourced.
Database Design. The database could be designed by an outside consultant who fully understood the processing requirements, an understanding which would take time and considerable effort. Because database administrator (DBA) activity would be ongoing for the life of the data warehouse, either the consultant would have to remain or the consultant should transfer the knowledge of how and why the database was designed, and this transfer should be made to a DBA employee. Database design could be outsourced, but only with a plan to make the full-time staff self-sufficient.
Extract, Transform and Load. The first part of the extract, transform and load (ETL) process is understanding the data sources, and this is best accomplished by an employee who knows these sources in depth. Bringing in a consultant requires the time and money to research the sources as well as the time required to ask questions of the company's employees. The transformations are organization-specific and best left to a knowledgeable employee. Integration is sometimes included in the ETL process; because it requires an in-depth knowledge of the organization and its business, it is not appropriate for outsourcing.
Data Cleansing. The analysis and processes for cleansing should be done by a knowledgeable employee, but the repetitive work of cleansing old data could be accomplished by an outsourcer.
Business Intelligence. Analysis work should not be outsourced because it requires a thorough knowledge of the data, the business, the management and what management needs. This should not be delegated to an outsourcer. However, creating some well-defined reports could be outsourced.
And then there's the issue of security. Outsourcing (not to mention offshoring) introduces additional exposures not found with internal employees. Mastering these exposures takes time and effort and reduces the financial benefits of outsourcing.
Management should recognize the wealth of information about the business that resides in the heads of existing staff. Even with excellent documentation (which few organizations possess), the gut sense, the domain knowledge and the context of how the business runs is not something that can easily be documented. Losing the staff that has this gut sense means losing the knowledge of the business and losing the ability to deal with the unforeseen situations that can only be addressed by people with a solid understanding of what's important. This capability will almost always be lost as an organization outsources. There is no way to transfer this understanding to the outsourcers. Recognize that the success of your organization is ultimately based on your loyal, existing staff, not on the outsourcer. Be sure to treat your staff with respect, real support, compensation and honesty. Also recognize that the success of the project rests with your organization, not with the outsourcer.
The impact on the morale of your existing staff from outsourcing could be devastating. Outsourcing will forever tar your organization as one that has little loyalty to its employees and, therefore, recruiting will be more difficult. No one on your staff will be thrilled with training his or her replacement, and you can expect lower productivity, lower quality, more absence, more disability claims, more grumbling by the water cooler and possibly sabotage. To block outsourcing, your staff can frustrate meetings with vendors and find many reasons to oppose the outsourcing effort. Asking your staff to participate in outsourcing will require candor, honesty and managers who are trusted. You may have trouble retaining disgruntled staff who are necessary to communicate with and train the outsourcer.
Management should carefully consider which, if any, of the roles in data warehouse creation and maintenance should be outsourced.