In the past five years, the data warehouse/data mart marketplace has grown from wishful thinking to reality. In the beginning, there were some concepts that were known as the architected DSS/data warehouse environment. To implement those concepts there were a few immature products. Today the data warehouse/data mart marketplace is flush with new and recycled products, and each new trade show brings even more products and innovations to the marketplace. Figure 1 is a brief depiction of the data warehouse/data mart marketplace segmentation as of the autumn/winter of 1997.
There are many and diverse segments to the data warehouse marketplace. The following are the different components and subcomponents of the data warehouse/data mart marketplace:
Hardware Platforms: There is a combination of MPP and SMP hardware platforms for the data warehouse environment. Powerful workstation technology and low-end client/server processors serve the data mart and analytical workstation portion of the marketplace.
DBMS: The basic DBMS technology that serves the data warehouse marketplace is almost exclusively relational.
Data Movement Products: Data movement products have proliferated faster than any other segment of the marketplace, to the point that the products have split into sub-markets. Data movement products are those products that search the different legacy applications and move the data that has been selected into the data warehouse. The different subsections of the data movement marketplace are:
High Speed Movement: High speed hardware/software is technology that is geared for the very rapid transfer of data from one technology to another. In this segment of the marketplace, the emphasis is almost entirely on the speed of movement. Very little, if any, integration or conversion occurs during the movement of data.
Simple Extraction: Simple extraction is software that is optimized on the simplicity and elegance of the end-user interface. The simple extraction products look easy to use but are unable to handle large volumes of processing and are unable to do complex integration and transformation conversions.
Product-to-Product Movement: These software products specialize in the movement and conversion of data from one product to another. In some cases, these products can handle large amounts of records. Very little significant integration and transformation is done by this kind of software.
Integration and Transformation Software: Integration and transformation software is designed for industrial-strength transformation of data. Everything but the most complex conversions is handled in an automated manner. Because of the robustness of the transformation logic that is supported, code is created.
Scheduling Software: Scheduling software allows the transformation programs to be scheduled and executed in an automated manner.
Data Warehouse Meta Data Management: Data warehouse meta data management software collects, stores, organizes and disperses meta data for the data warehouse environment. This software organizes the data and has automated interfaces to and from many types of software.
Data Mart Extraction: Data mart extraction software moves data from the data warehouse to the data mart environment. Data mart extraction software assumes that the data being read is already integrated. The software typically summarizes, merges and customizes the data prior to entry into the data mart.
Simple Analytical Processing: Simple analytical processing software performs simple analytical and spreadsheet processing.
Data Mining: Data mining software looks over detailed data for important business patterns and relationships.
Multi-dimensional Database Processing: Multidimensional database software creates "cubes" of data that can be analyzed in many ways. Usually the cube contains numeric data. The data inside the cube must be heavily and rigidly preprocessed before it can be entered into the cube. Once into the cube, the data can be analyzed in many different ways.
OLAP: Data mart processing. The data contained here is customized to meet the needs of the department that it serves.
Data Mart Meta Data: Meta data is attached or collocated at the server that manages the data mart. The meta data supports drill-down processing, describes the interface to the data warehouse and identifies the data in the data mart itself.
CASE/Data Modeling: CASE/data modeling software supports the design and modeling needs of the data warehouse/data mart environment.
DSS Systems Management: This category of software is one of the fastest growing. There are several significant sub-categories:
The Data Warehouse/Data Mart Marketplace
Resource Governor: This software attempts to enhance performance by dynamically limiting the resources used by a transaction without using too many systems resources.
Activity Monitor: This software keeps track of the processing done by the end-user community and identifies dormant data that can be safely removed from the data warehouse and hot spots that deserve special tuning and design consideration.
Data Monitor: This monitor looks at the content of data and assesses such things as the quality of data inside the data warehouse and the specific parts of the warehouse that are growing.
Refreshment Tracker: This software tells the end user exactly when a table has been loaded so that the end user can use the most current data.
Data Cleansing: Data cleansing software "scrubs" legacy and raw data before entry into the data warehouse. Some data cleansing software uses artificial intelligence techniques in order to achieve the best results.
Changed Data Capture: For many kinds of processing, log or journal tapes are a much better source than reads of the native legacy DBMS. But reading log tapes is not easy. Thus, utilities exist to pull desired data from the log tape and format the data prior to updating the data warehouse.
ODS Platform Technology: ODS requires a different set of hardware parameters in order to be successful. Most ODS operates optimally on an MPP configuration.
The data warehouse/data mart marketplace is remarkable in several ways: it is very diverse, it has been created very quickly, and it continues to grow and diversify at a stunning pace. Additionally, it is made up of many vendors. There is no such thing as a single vendor for the many aspects of the marketplace. There is a line of thought that states that one-stop shopping would be desirable. But that simply is not the case.
Why is there such a plethora of products? There are several reasons.
The data warehouse/data mart marketplace simply evolved the way that it did without the guidance or leadership of any given vendor. The marketplace is not a vendor- driven marketplace.
When one looks at the entirety of the products that is represented in the data warehouse/data mart marketplace, the number of specialized and diverse technologies is incredible. Therefore, it is not a surprise that many different companies, each with their own core competencies, are represented.
But there is perhaps another reason why there are so many companies, and that reason is that the data warehouse/data mart marketplace has been truly a consumer-led marketplace. Unlike marketplaces of the past, where products drove consumer demand, this marketplace is very different.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access