Data Quality, Data Synchronization and Integration Challenges
Generally speaking, data has a tendency to "decay" and become stale with time as changes to the environment at large occur on a regular or sporadic basis (e.g., products change names, financial systems change accounting rules, etc.). This is especially true for customer data. Indeed, a 2003 study completed by The Data Warehousing Institute entitled "Data Quality and the Bottom Line" stated: "The problem with data is that its quality quickly degenerates over time. Experts say two percent of records in a customer file become obsolete in one month because customers die, divorce, marry and move."
To put this statistic into perspective, assume that a company has 500,000 customers and prospects. If two percent of these records become obsolete in one month, that would be 10,000 records per month or 120,000 records every year. So, in two years, about half of all records are obsolete if left unchecked.
Customer data integration systems can create and maintain a consistent, accurate, and complete customer view. Critical customer relationship decisions such as promotions, price changes and discounts, marketing campaigns, credit decisions and daily operations revolve around key customer data. Without an accurate and complete view of the customer, efforts to provide targeted, personalized and compelling products and services may prove to be ineffective.
In no small measure, a company's success or failure is based on the quality of customer information. The challenge of CDI is to be able to quickly and accurately capture, standardize, and consolidate the immense amount of customer data that comes from a variety of channels, touchpoints and application systems.
Many organizations have separate sales, operations, support and marketing groups. If these groups have different databases of customer data - and different methods for recording and archiving this information - it is extremely difficult for the enterprise to rationalize and understand all of the customer processes and data infrastructure issues simultaneously.
The crux of the problem most companies face is the inability to compile a complete customer view when most of the systems are isolated from each other in their stovepipes and operate independently. By definition, a CDI system effectively bridges the gap between various customer views by rationalizing, integrating, and aggregating customer data collected from disparate data sources and applications in order to provide a single, accurate, consolidated view of the customer.
CDI systems pull critical customer information from existing internal and external data sources and validate that the data is correct and meets the business needs and data quality standards of the enterprise. Over time, CDI solutions can further enrich customer data with additional internal and external information, and store, manage, and maintain the customer master data as an authoritative system of record for the enterprise.
>One approach for maintaining data integrity would be to attack the problem at the operational system level. This seems to be a practical approach. After all, operational systems support applications that manage and execute transactions and maintain transactional properties of atomicity, consistency, isolation and durability (ACID). To sustain these properties, operational, transactional applications tend to be isolated from nonoperational applications such as data warehousing and business intelligence systems. This "isolation" helps discover and resolve data integrity issues one system at a time.
Correcting data quality is only part of the problem. Once the data has been corrected, the data changes have to be synchronized and integrated with various data sources. Moreover, various data records about the same individual need to be aggregated into a single customer-centric database. Although data integration challenges are not new and are not unique to the CDI space, customer data integration emphasizes their criticality. The challenges of data integration include the following:
- Lack of standardization of customer or company names and addresses Without standardizing this information, it is difficult to resolve customer lifetime value, as customers may have different representations within databases.
- No common identifier or linking of customers across systems For example, an individual customer record may be stored in several operational systems, and thus the customer can be represented differently in every system, even using different names and aliases. This representation mismatch may prevent a CDI solution from recognizing the same individual across various applications and data stores.
- Incorrect data Traditional customer data solutions often use special codes to signify unknown or default data. For example, a phone number of "999-999-9999" or a birth date of "01/01/01" may represent common shortcuts for unspecified or missing data. These special codes may have to be treated in a way that escalates data content questions to the appropriate data steward.
- Stale, outdated data. As we stated earlier, data has a tendency to change over time. Left unchecked and unmanaged, these changes do not get reflected in the customer database, thus significantly reducing the value of data.
One of the stated goals and key requirements of any CDI system is to create and maintain the best possible information about customers regardless of the number and type of the source data systems. To achieve this goal, Customer Data Integration should support an effective data acquisition process. The CDI process requires different steps and rules for different data sources. However, the basic process is consistent, and at a minimum should answer the following questions:
- What points of data collection might have customer information?
- How does each data source store, validate, and audit customer information?
- What sources contain the best customer data?
- How can data be integrated across various data sources?
- What information about customers is required for current and future business processes?
- Where does this customer information reside?
