Continue in 2 seconds

Straight Talk: Data Warehousing and Offshore Development

  • November 01 2004, 1:00am EST

Two phenomena are occurring simultaneously today - the advent of corporations building data warehouses and the movement of development of IT projects offshore. It is therefore a natural question to ask: Are we going to be developing data warehouses offshore? However, there are many good reasons why offshore development of data warehouses is not in the cards.

Why is it that data warehouses do not lend themselves to offshore development? Why are data warehouses different from other types of IT projects and structures?

From a tactical perspective, data warehouses deal with and operate on data, not processes. It is processes that are economically built offshore. Data is a different story altogether. Whereas you can specify processes down to a fine degree, specifying, defining and developing data for a data warehouse is a different beast altogether.

Another factor making data warehouses poor candidates for offshore development is that there is no such thing as a generic data warehouse. Each data warehouse is different for each company - even companies in the same line of business. You cannot make cookie-cutter data warehouses.

Another critical tactical reason why you should not develop data warehouses offshore is that the very nature of data warehouses is an environment in which requirements are never firm. Data warehouses are built for end users. Additionally, end users operate in a mode of not knowing what they want. End users operate in a mode of, "Give me what I say I want, then I can tell you what I really want." They also operate in a mode of discovery. They say, "Ah - now that I see what the possibilities are, I can tell you what I really want. However, until I can see what the possibilities are, I really can't tell you what I want." This very fluid development environment does not lend itself to offshore development because offshore development is geared for very rigid requirements.

There is also another very different set of motivations for not building a data warehouse offshore, stemming from the fact that a data warehouse is an essential architectural component of the corporation. Once the data warehouse is built, various other processes emanate from it. The data warehouse becomes the hub for many other kinds of processing including data mining and exploration, data marts, operational data stores, decision support system applications and project marts. Once built properly, the data warehouse becomes the central nervous system for the intelligence of the corporation. Therefore, the data warehouse must be built architecturally in a very proper manner. Some of the more important architectural considerations of a data warehouse include:

Size. If my data warehouse starts to grow beyond approximately ten terabytes, the cost of the infrastructure becomes a burning issue. Therefore, data warehouses must be built so that they can support very large amounts of data without the cost of the infrastructure obviating the benefits of building the data warehouse in the first place.

Unstructured data. One of the advantages of a data warehouse is it can support unstructured data as well as structured data. However, the warehouse must be built properly in order for unstructured data to be properly integrated.

Data mining and exploration. One of the long-term benefits of a data warehouse is that it can support data mining and data exploration. Again, the warehouse must be built properly from the outset in order to be able to take advantage of these kinds of technology.

ODS. One of the most important types of architecture is often the operational data store (ODS). The ODS and the data warehouse need to cohabitate in a complementary and constructive manner. The lines between the ODS and the data warehouse must be constructed properly from the outset.

And there are many other architectural considerations for the building a data warehouse. The immediate set of end-user requirements are only one set of considerations - although certainly an important set. In the long run, the architectural considerations for the building of a data warehouse actually outweigh the immediate end-user requirements.

Therefore, there are many important architectural considerations that need to be factored into the building of a data warehouse. It is fair to say that offshore developers have not reached the levels of sophistication to be able to deal with the architectural considerations of the design and development of a data warehouse.

Another important strategic reason for not offshoring the development of a data warehouse is that - in a way - data warehouses are never complete. As long as the business is growing and as long as the world is changing, there will be a need for adding to or altering a data warehouse. Having the ability to alter the data warehouse in house is very important. Relying on an offshore source for the ongoing maintenance and development of a data warehouse is very risky business.

Yet another reason for not offshoring the development of data warehouses is that the security and privacy of the data is subject to compromise. Data warehouses contain corporations' bedrock data. Allowing that data to get out of control is taking a very serious risk, one where the very core business of the corporation may be compromised. Indeed, in some cases it may actually be illegal to build a data warehouse offshore because of the nature and content of the data.

In some ways, building a data warehouse is like having brain surgery. If a person must have brain surgery, he or she surely wants the brain surgery to be done correctly, where there are the best chances of a successful operation and recovery. You want an experienced brain surgeon, one who has seen abnormal conditions before and knows how to handle them. Having a rookie explore the brain as part of a learning experience introduces a level of risk that is unacceptable in most environments.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access