The craze for data mining--one of the most important, most talked-about technologies for today's businesses--is a relatively recent phenomenon.

This is despite the fact that data mining actually has deep roots in the fields of statistics and artificial intelligence (AI). So while AI systems have made significant advances over the years, in many cases it has remained an isolated technology that hasn't generated much excitement for people involved in sales and marketing functions.

This has all changed thanks, in part, to recent marketing strategies known as "customer management" or "relationship marketing." These tactics, in which corporations of all sizes take a highly focused look at the attributes and attitudes of their individual customers, have quickly piqued marketing executives' curiosity in data mining and database marketing applications.

Aside from the business factors driving the widespread appeal of these applications is, of course, the technology itself. While business trends and buzzwords such as "relationship marketing" may come and go (remember "business process reengineering" from the early 1990s?), it is undeniable that technology has made giant steps forward in helping businesses better understand and leverage their vast stores of data.

So what does this mean for the database and systems administrators who manage their companies' customer databases? If you want to come through for your company and support its new customer-focused database marketing efforts, you'll need to know how to build and manage the highly scalable data mining software systems that will help you and your company succeed.

Let's start by taking a look at some sample "real-life" marketing scenarios in order to better understand how these situations can drive the need for scalable data mining systems.

Scenario No. 1: The Need for Highly Focused Marketing Campaigns

For example, suppose your company's vice president of marketing comes to you and says:

"Our far-reaching marketing programs are not effective enough. We're devoting too much time and spending too much money on mailings, special events and advertisements that aren't bringing in new customers. Can you do anything to help?"

The problem here is that your company is spending too much money marketing to people who aren't purchasing the product. Marketing's need here is to identify your customers' unique attributes and look for new sales prospects in similar situations.

Scenario No. 2: Making the Most of Your Company's Customers

How about another example? This time sales are up. Marketing is on track. Business is great. But suddenly there are new players in your company's market, and your marketing executive's needs have changed:

"We've had some real success growing our customer base, but I think we've nearly reached our peak at gaining new customers. Let's see how we can generate additional sales with the customers we already have."

In this second scenario, your company's success has already provided you with an expanded customer base. Now instead of increasing market share, your company needs to increase its customers' "wallet share."

Scenario No. 3: New Strategies for Customer Retention and Acquisition

Tough competition plays a part in this final scenario as well. Now your company is losing retail market share to a competitor that has just opened shop down the street from your store:

"We need to put together a promotional campaign to bring back our old customers as well as a number of new ones. Let's find out our strengths compared to the competition and market these factors accordingly."

With this scenario, marketing will need to divide your customer base into segments--including drive time from store, household income, age group and educational levels--in order to understand in which segments your company has an advantage over the competition. Then your company can tailor new promotional campaigns for these individual customer groups.

Advanced Customer Knowledge

While traditional decision support tools are great for asking quantitative questions (such as "What are my average sales for the quarter?" or "Which products are selling better this year than last year?"), they don't work well for providing the complex, qualitative data that drives today's marketing efforts.

This is because traditional decision support tools require business analysts to first form a hypothesis upon which to base their marketing efforts and then use query tools to collect the information necessary to prove or disprove their hypothesis. This "verification-based" approach is flawed because it relies on the analyst to have the correct intuition about what the answer really is. If the analyst doesn't think of the correct answer, then it will never be found.

Data mining--the process by which a set of algorithms systematically looks through your data to automatically find patterns, trends and correlations--is a better tool for taking advantage of customer-focused data.

For example, a simple product marketing campaign might use only one data mining model to predict a customer's propensity to buy. But for today's advanced customer management and loyalty campaigns, your marketing team will probably require a more complete understanding of the customer, taking into account several additional factors.

A more sophisticated problem--such as in Scenario No. 2 where your company is looking to generate additional sales from its existing customer base--will not only look to a customer's propensity to buy, but will also require additional data mining functionality. Your marketing team may want to take into account other factors, such as this customer's channel preference, product affinities, fee tolerance and, most importantly, predicted customer lifetime value.

Only when viewed together can these different attitudes and attributes give a company a complete understanding of a customer and his purchasing tendencies. Some of these models mentioned above are data intensive, while others are computationally intensive. Both types benefit from the scalable power of parallel processing software systems that are tuned to take advantage of your server system's parallel architecture.

It's important to note that scalability isn't just about being able to solve big problems. It's about being able to solve problems that get bigger over time, regardless of the size at which they start. And these applications can grow rapidly in at least three dimensions: data size, functionality and complexity.

Scenario No. 3 provides a classic example of how companies' desires to understand very specific customer behavior can increase an application's complexity, thereby driving the need for scalable systems. For example, if you're trying to understand the purchasing patterns of customers within a certain geographical area, you have a certain amount of complexity involved. If you then become more specific and want to target only those customers living in that area within a certain age group, your complexity further increases and your application needs expand.

The Right Tools for the Job

Today's database marketing applications demand that you are skilled with the techniques required for building scalable applications on top of scalable technologies. A solid understanding of how to integrate these technologies is essential, so as to not introduce any bottlenecks into your application design.

For example, using data mining to develop predictive models (such as Scenario No. 1's "What types of consumers are most likely to purchase my company's product?") can be extremely computationally intensive, making this one area where the scalability of your data mining system becomes critical.

The algorithms involved in a sophisticated query can generate dozens, hundreds or sometimes thousands of models. In addition, you'll need to test the accuracy of these models against a significant portion of the database (at least 10 percent). Having a parallel, high-performance system is critical to ensuring fast response times for complex data mining queries, where marketing needs may dictate that some queries complete in minutes or even seconds.

Highly scalable hardware platforms such as massively parallel processing (MPP) systems are the solution of choice to avoid this problem, enabling users to effectively process customer data within a reasonable amount of time. As your company's marketing efforts become more advanced and its customer base grows, more processors can be added to the platform. Otherwise, the computationally intensive nature of these advanced database marketing campaigns will grind your non-scalable system to an abrupt halt.

Depending on your level of knowledge of the latest technologies and techniques--combined with your ability to actually set aside the time to implement them--a better solution may be to work with a knowledgeable IS professional services firm with experience related to your industry's specific needs.

Look especially for firms with real-world experience building scalable solutions for large organizations. Ask around. Learn what companies have training and success in all phases of an application life cycle: requirements definition, architectural definition and final implementation. The right partner will be your most important ally, so don't be reluctant to shop around and ask questions.

After all, when the right technologies are combined with the appropriate techniques, the result will be a data mining application that meets not only your marketing department's current needs, but one which can satisfy even your company's most demanding requirements down the road.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access