Continue in 2 seconds

The DW/DM Marketplace

Published
  • January 01 1998, 1:00am EST

In the past five years, the data warehouse/data mart marketplace has grown from wishful thinking to reality. In the beginning, there were some concepts that were known as the architected DSS/data warehouse environment. To implement those concepts there were a few immature products. Today the data warehouse/data mart marketplace is flush with new and recycled products, and each new trade show brings even more products and innovations to the marketplace. Figure 1 is a brief depiction of the data warehouse/data mart marketplace segmentation as of the autumn/winter of 1997.

There are many and diverse segments to the data warehouse marketplace. The following are the different components and subcomponents of the data warehouse/data mart marketplace:

Hardware Platforms: There is a combination of MPP and SMP hardware platforms for the data warehouse environment. Powerful workstation technology and low-end client/server processors serve the data mart and analytical workstation portion of the marketplace.

DBMS: The basic DBMS technology that serves the data warehouse marketplace is almost exclusively relational.

Data Movement Products: Data movement products have proliferated faster than any other segment of the marketplace, to the point that the products have split into sub-markets. Data movement products are those products that search the different legacy applications and move the data that has been selected into the data warehouse. The different subsections of the data movement marketplace are:

High Speed Movement: High speed hardware/software is technology that is geared for the very rapid transfer of data from one technology to another. In this segment of the marketplace, the emphasis is almost entirely on the speed of movement. Very little, if any, integration or conversion occurs during the movement of data.

Simple Extraction: Simple extraction is software that is optimized on the simplicity and elegance of the end-user interface. The simple extraction products look easy to use but are unable to handle large volumes of processing and are unable to do complex integration and transformation conversions.

Product-to-Product Movement: These software products specialize in the movement and conversion of data from one product to another. In some cases, these products can handle large amounts of records. Very little significant integration and transformation is done by this kind of software.

Integration and Transformation Software: Integration and transformation software is designed for industrial-strength transformation of data. Everything but the most complex conversions is handled in an automated manner. Because of the robustness of the transformation logic that is supported, code is created.

Scheduling Software: Scheduling software allows the transformation programs to be scheduled and executed in an automated manner.

Data Warehouse Meta Data Management: Data warehouse meta data management software collects, stores, organizes and disperses meta data for the data warehouse environment. This software organizes the data and has automated interfaces to and from many types of software.

Data Mart Extraction: Data mart extraction software moves data from the data warehouse to the data mart environment. Data mart extraction software assumes that the data being read is already integrated. The software typically summarizes, merges and customizes the data prior to entry into the data mart.

Simple Analytical Processing: Simple analytical processing software performs simple analytical and spreadsheet processing.

Data Mining: Data mining software looks over detailed data for important business patterns and relationships.

Multi-dimensional Database Processing: Multidimensional database software creates "cubes" of data that can be analyzed in many ways. Usually the cube contains numeric data. The data inside the cube must be heavily and rigidly preprocessed before it can be entered into the cube. Once into the cube, the data can be analyzed in many different ways.

OLAP: Data mart processing. The data contained here is customized to meet the needs of the department that it serves.

Data Mart Meta Data: Meta data is attached or collocated at the server that manages the data mart. The meta data supports drill-down processing, describes the interface to the data warehouse and identifies the data in the data mart itself.

CASE/Data Modeling: CASE/data modeling software supports the design and modeling needs of the data warehouse/data mart environment.

DSS Systems Management: This category of software is one of the fastest growing. There are several significant sub-categories:

The Data Warehouse/Data Mart Marketplace


  1. Hardware Platforms: The hardware that the data warehouse resides on
    • MPP ­ Massively Parallel Processor
    • SMP ­ Symmetric Multi-Processor
  2. DBMS: The basic data management software that stores, retrieves and indexes the data in the data warehouse
  3. Data Movement: Software responsible for transporting data from the legacy environment to the data warehouse or ODS environment
    • High Speed ­ Hardware/software whose primary focus is on the rapid movement of data from one environment to another; very little or no integration and transformation is done
    • Simple Extraction ­ Tools whose goal it is to simplify the interface for the specification of what data is to be moved where; very little or no integration and trans- formation is done
    • Product-to-Product Movement ­ Software that opti- mizes the movement of data from one technology to another; very little integration and transformation is done
    • Integration and Transformation ­ Software whose goal is to convert and transform data as it is moved into the ODS or the data warehouse
    • Scheduling Software ­ Software that tracks the progress of the execution of a data movement job
  4. Data Warehouse Meta Data Management: Software whose job it is to collect, organize, assimilate and prepare for display meta data for the data warehouse
  5. Data Mart Extraction: Software whose job it is to summarize and customize data found in the data warehouse environment
  6. Simple Analytical Processing: Software whose job it is to do very basic analytical processing
  7. Data Mining: Software for explorers designed to find patterns and business relationships that have not previously been discovered
  8. Multidimensional Processing: Cube processing along different dimensions
  9. OLAP: Data mart or departmental processing
  10. Data Mart Meta Data: Meta data management for the data mart
  11. CASE/Data Modeling: Software for capturing and documenting the enterprise view of information systems
  12. DSS Systems Management: Software for management of the DSS environment
    • Resource Governor ­ Software to terminate queries based on resource consumption
    • Activity Monitor ­ Software to find dormant data and "hot spots"
    • Data Monitor ­ Software to determine data quality and to prepare for capacity planning
    • Refreshment Tracking ­ Software for telling the end user how fresh the data is in the data warehouse
  13. Data Cleansing: Software for scrubbing data prior to initial entry into the data warehouse
  14. Changed Data Capture: Software for reading of the log and journal tapes preparing the transaction data for entry into the data warehouse
  15. ODS Platform: Hardware for managing the mixed workload of the ODS environment

Resource Governor: This software attempts to enhance performance by dynamically limiting the resources used by a transaction without using too many systems resources.

Activity Monitor: This software keeps track of the processing done by the end-user community and identifies dormant data that can be safely removed from the data warehouse and hot spots that deserve special tuning and design consideration.

Data Monitor: This monitor looks at the content of data and assesses such things as the quality of data inside the data warehouse and the specific parts of the warehouse that are growing.

Refreshment Tracker: This software tells the end user exactly when a table has been loaded so that the end user can use the most current data.

Data Cleansing: Data cleansing software "scrubs" legacy and raw data before entry into the data warehouse. Some data cleansing software uses artificial intelligence techniques in order to achieve the best results.

Changed Data Capture: For many kinds of processing, log or journal tapes are a much better source than reads of the native legacy DBMS. But reading log tapes is not easy. Thus, utilities exist to pull desired data from the log tape and format the data prior to updating the data warehouse.

ODS Platform Technology: ODS requires a different set of hardware parameters in order to be successful. Most ODS operates optimally on an MPP configuration.

The data warehouse/data mart marketplace is remarkable in several ways: it is very diverse, it has been created very quickly, and it continues to grow and diversify at a stunning pace. Additionally, it is made up of many vendors. There is no such thing as a single vendor for the many aspects of the marketplace. There is a line of thought that states that one-stop shopping would be desirable. But that simply is not the case.

Why is there such a plethora of products? There are several reasons.

The data warehouse/data mart marketplace simply evolved the way that it did without the guidance or leadership of any given vendor. The marketplace is not a vendor- driven marketplace.

When one looks at the entirety of the products that is represented in the data warehouse/data mart marketplace, the number of specialized and diverse technologies is incredible. Therefore, it is not a surprise that many different companies, each with their own core competencies, are represented.

But there is perhaps another reason why there are so many companies, and that reason is that the data warehouse/data mart marketplace has been truly a consumer-led marketplace. Unlike marketplaces of the past, where products drove consumer demand, this marketplace is very different.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access