When considering the most important factors in implementing a data warehouse, many organizations address cost, schedule priorities, hardware platforms, DBMSs, business intelligence tools, population tools, etc. While all of these components are important, the organization of the warehouse implementation team(s) should be considered at least as important. In many cases, the implementation teams are simply thrown together in the most expedient manner, without a lot of thought toward how they will all function. With a little foresight, and without making massive structural changes to the IT organization, it is possible to set up a warehouse implementation team that will maximize the chances of a successful implementation. In the early data warehouse days, the vision was of an enterprise-wide data warehouse satisfying all the DSS requirements of the company. Many IT organizations set up a data warehouse group, whose job it was to model and build the data warehouse, regardless of delivering information to the end users. These groups designed the warehouse and warehouse population processes, with major responsibilities for data modeling, determining the best sources of data in the legacy environments, extracting, cleansing and loading the warehouse. Functional delivery groups who had traditionally supported the various user groups did the job of delivering the information to the end users. This led to warehouse development teams that were primarily organized functionally, e.g., a team for sales, another for marketing, another for finance, etc. Inside each of these "vertical projects," the team was made up of several different organizations (e.g., the business user, the information delivery organization, the warehouse developer and the DBA) with roles and responsibilities clearly delineated. This team functions in a "customer/supplier" role, with the business user as the customer of the information delivery specialist who is, in turn, a customer of the warehouse developer. While this structure allows individuals to focus on their specialties, it can produce barriers and finger-pointing between the different IT areas, as well as a tendency to "throw each deliverable over the wall" to the next group in line. For example, it is easy to blame other groups when projects start slipping. We have often heard, "Well, I got the access layer working on time, but the population guys aren't ready with the populated databases yet." See Figure 1.
Figure 1: Functional Data Warehouse Development Teams
As the enterprise data warehouse fell out of grace in the mid-1990s to be replaced with the rush to the data mart strategies being touted by many vendors, the organizational issues became more clear. If there was no requirement for an enterprise data warehouse, it was difficult to see why there would be a requirement for a data warehouse group. Many organizations downsized or eliminated their data warehouse group in favor of data mart teams, focusing on single implementations. If there was any central warehouse group at all, its function had become diminished to setting standards, providing direction and other largely ceremonial duties. All the interest was focused on the data mart teams.
With the more recent realization that this data mart strategy worked well for individual groups of users but did little to address the needs for a common view of information across the company, many organizations are now promoting a blended strategy of "enterprise data warehouse by way of cooperative data marts." This approach combines the speed of delivery promised by implementing data marts with the ability to provide a common view of cross-functional data touted by the enterprise data warehouse. In order to ensure the success of this latest approach, we should learn lessons from the organizational mistakes of the previous strategies.
First, let us look at the makeup of the individual data mart, or subject area, delivery teams. A successful organizational structure used to implement warehouse projects is one in which the team is regarded as a single, cross-functional entity with the objective of satisfying a specific business requirement. Rather than the project team members regarding themselves as primarily function-driven, the team consists of participants who may fulfill one or more roles. The roles are defined in four disciplines: business, data, application and technology. Although this may be seen as organizing functionally again, the roles are not meant to define responsibilities and may be filled by different people at different times. Additionally, one person may function in more than one role at a time. For example, in the traditional model, an information delivery analyst will interview a business user for requirements and document them for passing on to application developers in both the information delivery organization and the warehouse development organization. In the new model, the business subject-matter expert (SME) may fulfill the role of user and analyst simultaneously and work directly with the development team during prototyping of both the business intelligence and warehouse-built applications. In this way, communication is improved, and the business application has built-in approval at each stage of development. Note that fulfilling one or more roles in a warehouse development project has no bearing on where a particular person fits organizationally. This team may be described as a "business-driven work group structure." Figure 2 illustrates the cross-functional work group concept.
Figure 2: Cross-Functional Work Group
This project team structure can be formed regardless of where each team member fits organizationally. In other words, the underlying IT organizational structure can still exist as before, with people being assigned to delivery teams according to the requirements for a specific skill. This type of structure has been used successfully for many years for support roles such as DBAs, where a single DBA may have responsibility for many projects (some in delivery mode and some in maintenance) and splits his/her time according to requirements.
The project organization previously described will suffice for a single data delivery project. However, in building a cross-functional warehouse, there will necessarily be multiple delivery projects, some developed serially and some in parallel. In order for the common, cross-project (and cross-functional) issues to be successfully resolved, a further component needs to be in place: a coordination function a warehouse management office (or WMO).
This WMO will work with each development project to coordinate resources and dependencies. It should also perform a warehouse coordination role for data and processes that have already been implemented, along with a role to define and enforce enterprise data warehouse standards and guidelines. These standards and guidelines should include warehouse architecture and methodology at a minimum, with additional responsibilities for defining tools and maintaining vendor relationships as required.
In its development role, the WMO should help each project understand and implement the architecture and methodology standards and guidelines. From a methodology perspective, it helps with the development and administration of a project plan. As these plans are implemented, the warehouse management office gathers key status information across projects and builds a comprehensive picture of all data warehouse activities. From this information, the warehouse management office can identify key project risks and synergy, and the WMO has the ability to develop plans to mitigate the risks and maximize the synergy. This role will enable the WMO to coordinate changes across projects:
- Business process definition can be coordinated when two or more projects will impact a single organization.
- Application development can be coordinated when two or more projects include similar functionality, leading to a library of reusable application objects.
- Technology changes can be coordinated to leverage vendors.
- Data changes can be coordinated to ensure a single view of data across the enterprise.
- Organizational changes can be leveraged across similar organizations.
- Location can be coordinated when more than one project needs to roll out to the same facility.
The warehouse management office will also ensure that common cross-functional data models and definitions are used within the warehouse. It will set and maintain standards and procedures for the technical implementation of the delivery projects and be responsible for ongoing administration of the warehouse, decisions on architecture and infrastructure components, and changing standards. One particularly important role that the WMO can fill is that of ensuring common definitions of the cross-functional dimensions, or conforming dimensions, as described by Ralph Kimball. If there is no organization to fill this role, it becomes impossible to view data from the same perspective across data mart implementations. As well as the definition of common dimensions, the WMO will be able to identify occasions when code and processes will be able to be reused across projects, as well as cases where similar projects can use the same technical infrastructures.
The WMO should be used to provide a home for the data warehouse core competency. In this role, it will supply a representative to each development project team a warehouse architect. This representative will have a twofold responsibility: 1) Ensuring standards and guidelines are followed, and 2) increasing the chances of success for the project by supplying data warehouse expertise and knowledge of reusable components to the development team. See Figure 3 for a depiction of the proposed structure. Please note that although the depiction shows that all members of the WMO are allocated to development teams, this will not always be the case. For example, some warehouse management office members will be concerned with strictly cross-functional warehouse issues, rather than specifics of a particular business information delivery project.
While not all of the functions described will be necessary for every instance of a warehouse management office, companies may choose which ones they need to implement.
The structure discussed in this article has the benefit of allowing each team to concentrate on delivering the maximum functionality to their users while still allowing the goals of cross-functional data viewing and enterprise data management to be realized. Without the implementation of such an organization, it is difficult to see how individual data mart implementations will be able to be integrated into an enterprise data warehouse.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access