Every year around this time, Raab Associates updates its Guide to Customer Matching Systems. After changes for each vendor have been posted, we have a rare opportunity to step back and look for larger patterns. Here's what we saw.
Search-based technologies. Most of today's commercial customer matching systems (such as those from Firstlogic, Group 1, Trillium, Innovative Systems and DataMentors) were originally designed to merge records into mailing lists or customer databases. However, another set of products has now appeared, with roots in searches against police, medical or immigration files. This set of products includes those from Search Software America, Intelligent Search Technology and ChoiceMaker. Like merge-oriented systems, the search products ultimately compare records to determine whether they match. However, the search systems generally apply more sophisticated mechanisms when selecting records to compare and are less dependent on parsing and standardization for accurate results.
The search systems are noteworthy for considering the frequency of individual names within their logic. This means they require more precise matches against more common names. Frequency is most often used during the initial selection of records to submit to a detailed matching process. This reduces the number of records that must be matched against common names (an important consideration for online search systems) without excessively limiting the candidates compared to unusual names. Frequency is also sometimes used within the matching logic itself, on the theory that unusual names are more likely to be misspelled, misformatted or otherwise mangled during data capture and processing. (SAS/DataFlux and Ascential QualityStage also use frequency in these ways.)
To reduce their reliance on parsing and standardization, search systems may match against all words in a name field, rather than attempting to identify family versus given names, or match against both original and standardized addresses rather than the standardized address only. This typically requires storing several versions of each reference record – another way to improve online response time, although at the cost of higher storage and initial processing expenses.
Search system vendors claim that their systems are more accurate and easier to set up than those of their merge-oriented competitors. Of course, the competitors disagree. Both point to tests that support their position, but the painful truth is that results will vary greatly depending on the data, applications and technicians involved. Careful buyers will run tests of their own.
Surveillance applications. It's old news that customer matching systems are now connected to online operational systems to identify customers during order entry, technical support and customer service transactions. Such applications came naturally to search systems, while merge- oriented systems added real-time candidate selection and matching against reference databases to support them.
These capabilities have, in turn, enabled new applications to spot fraud, money laundering and terrorists. These are based on existing matching functions but still require product extensions for new user interfaces, audit trails and reference list maintenance. Some also need new matching methods to find relationships among individuals and organizations. Surveillance is a new business for many matching vendors and often involves buyers who are not their traditional customers; however, several firms report that it already accounts for a substantial revenue stream.
Better handling of international data. Both customer files and surveillance systems must increasingly contend with names and addresses of people from outside the United States. These often involve spellings, rules and formats that are quite different from U.S. or Western European standards. Most matching systems can accommodate these names and addresses by applying specialized rules and data tables to their existing processes. While this is still a substantial investment, major vendors are increasingly finding it worth making.
Easier system configuration. Matching has traditionally been part of business processes, such as loading a data warehouse, that are set up and managed by technical users. The matching system itself would often be tuned once, during implementation, by specialists from the vendor or an outside consultancy. As a result, buyers focused on system accuracy and flexibility but paid little attention to ease of deployment or revision.
Today, matching systems are used for more applications and requirements can change quite frequently. Companies also have less time and money for implementation and maintenance. Vendors have responded by adding graphical user interfaces, reports and other features to speed and simplify deployment. In theory, this could allow non-technical users to set up their own processing streams and business rules.
Expansion beyond name and address matching. Matching systems are increasingly being asked to parse, standardize and match non-name and address data such as product IDs. This reflects the continued extension of business intelligence systems and the greater familiarity of technology departments with matching system capabilities. Streamlined configuration processes and matching methods that work despite minimal parsing and standardization are important contributors to this trend. Tight budgets, which make it more difficult to purchase new tools than it is to repurpose existing ones, may also have something to do with it.
Extended product suites. Several matching vendors have extended their products to provide input profiling, transformations, loading, consolidation, enhancement and analysis. Combined with the mix of batch and real-time processing capabilities, this allows them to position the systems as a "complete data quality solution" – with the precise definition of "complete" always conveniently corresponding to the particular set of features the vendor happens to offer.
This type of expansion is typical of the software industry, particularly for specialized products that become more widely adopted. The fundamental challenge faced by vendors in this situation is that once their tools become well understood, larger vendors of related products can easily add some version of the specialized capability and then squeeze the specialists into an ever-shrinking "best-of-breed" niche.
Line extension is therefore essentially a defensive move. The specialists seek a large enough presence to justify remaining part of corporate infrastructures or, failing that, enough revenue from a smaller number of clients that will enable them to survive. Historically, such expansion has rarely succeeded: small software companies have a hard time supporting a large product line, and big competitors have an overwhelming marketing advantage. Therefore, the strong likelihood is continued consolidation of the matching market as small companies drop out or, more likely, are acquired by larger firms for their technology. In other words, the changes we saw this year are just a hint of the changes yet to come.
David M. Raab is a Principal at Raab Associates Inc., a consultancy specializing in marketing technology and analytics. Raab is also the author of the book The Marketing Performance Measurement Toolkit (Racom Communications). He can be reached at email@example.com or via his blog at http://customerexperiencematrix.blogspot.com/.