Continue in 2 seconds

The End of Information Overload

Published
  • September 01 2005, 1:00am EDT

Marketing is finding that its once cozy role as the artists within the hard-nosed commercial world is being questioned with ever-increasing vigor. Management expects tangible results, wants more than just awareness campaigns and is after hard cash and a justified return on the dollars they are spending.

While most enterprises claim a degree of science behind corporate decisions, the truth remains that in what are still essentially product-centric silos, most production is decided on the basis of the intuition of the management team. When that fails and business results do not match expectation, the result is usually to sack the management and find others who the shareholders believe will guess better.

The better-organized enterprises will employ historic reporting to give them more detailed pictures of what has happened in the past, and that factual data will be used to frame the decisions. This tends to be fine until business change becomes more rapid than the ability of historic analysis to reflect, because the vast number of variables interacting and changing overwhelm the ability of mangers to analyze and synthesize in a meaningful fashion. At this stage, it becomes imperative to manage by predictive modeling, where analysis of the data enables the variables to shape the future with far more certainty than an educated guess.

A predictive model is a mathematical description to predict the future, in terms of a behavior or outcome, based on an existing data pattern. The model is built using known outcomes to train the model to identify the conditions that typically contribute to the result. Using mathematical models rather than intuition means that the explanation of the model is far more transparent. In a fast-moving world, it then becomes far more obvious what is changing and how that change is impacting outcomes. The models will not only identify what is important to an outcome, but will also assign value to their impact on the outcome. Therefore, instead of having hundreds of individual drivers, attention can be focused on those with the largest impact.

The New CRM Paradigm

It is now increasingly recognized that the aim of customer relationship management (CRM) must be to understand a customer's impact on business performance - to determine how best to communicate to customers in order to cost efficiently and effectively enhance the level of their current performance, be that by staying longer, buying more or buying things with a higher net margin. All of this is undertaken within a framework of managing customer lifetime value, so parochial end-of-quarter sales decisions should not endanger the overall value of an account. The analysis of all of the facts within the CRM systems captured and fed to the data warehouse needs to be readily converted into knowledge and returned to the CRM systems to influence and coordinate behavior.

In the new paradigm, advanced analytics become essential. However, to date the means of undertaking that analysis has often been at odds with the business reality. There are few organizations with the required levels of trained statisticians and data analysts to support the business. Few of those trained as statisticians or data analysts actually understand the business in sufficient detail to make best use of their skills, and the business domain experts are too uncomfortable with the arcane and opaque nature of analysis to be able to employ the techniques themselves or readily understand some of the more obscure outputs when presented by the analytics professionals.

While the current popular predictive analytics platforms have made strides in presenting information through mediums such as Internet portals, the techniques themselves still create many problems. Traditional methods are still largely reliant on an installed base of trained and skilled advocates who, while important in advancing the case for analytics, also act as barriers to any fundamental changes to their elite status as the masters of the arcane dark arts of modeling and analysis.

New tools are now available that are designed for use by both the trained and the untrained statistical user. These tools automate the process converting data into knowledge and provide results that are essentially far more intuitive for the non-statistician to understand. For the trained modeler, it is a productivity aid, whether used in its own right or in support of other traditional techniques (such as neural networks) where it can be used to identify the variables to be used in a model quickly and reliably rather than needing to use trial and error.

The Demand for Change

Traditional statistical techniques were developed to enable small samples to be analyzed. From the findings within the sample, it was then possible to generalize about a larger population. The key to the process of generalization was to make certain assumptions about the shape and the distribution of the data - and if the data was skewed away from the desired shape, to adjust it mathematically so that technique would work correctly and the generalization would be robust.

However, in business we often do not want to deal with a small sample; we actually want to analyze the full population. Much skill is then required to manipulate the data to ensure that it is distributed correctly for the technique to be applied soundly, and these are skills that are clearly not common among business experts.

The technique that is employed when building a predictive model is to split the data in two. One set is the training set, and the other is the validation set. We know the outcomes for both sets. The training set data is used to train the model to identify the relevant outcome. The validation set is then used to check that the model is able to identify the correct outcomes in another set of data. This ensures that the model is not so overly adapted to the training set that it cannot handle other data. This is a process of iteration with the process passing through loops of modify, model, test, modify, model and test to avoid what is termed an over-fitted model.

This process of iteration requires skill to create a suitably optimized balance between predictive power and general applicability or robustness. Today, a new breed of predictive analytics tools, based on structured risk minimization (SRM), automates this process, removing the need for highly skilled operation and the time taken to loop through the process. The data does not need to be manipulated to adjust its distribution; the technique uses the values of the data to measure their relative impact on the question to be answered.

SRM-based analysis is able to produce a robust model, without the need for expert skills and with a more meaningful answer being produced incredibly quickly. The tool itself balances and reports on the accuracy and reliability of the answer without the need for repetition at the hands of the skilled analyst.

Vladimir Vapnik, the mathematician credited with creating SRM, worked on the issues surrounding the analysis of very large datasets without the need to make assumptions about the distribution of the data.

The requirement is to make a model that is not so complicated that it cannot be repeated, nor so simple that it fails to be informative. The solution is to measure the complexity of the model and use that as a controlling function in the data analysis. As the information fed to a traditional model increases, the training error decreases and it becomes more predictive. At the same, time the confidence interval increases, and it becomes less robust in its ability to handle new data.

Vapnik's breakthrough was to find a way to calculate the confidence interval. For any given complexity of a given set of data, it becomes possible to measure the ability of the model to handle generalization. The advantage of SRM is that it automatically produces robust reliable models very quickly and without the need for manual intervention to adjust the data to produce normally distributed data.

For the business user, the benefit is that new SRM-based tools enable the non-statistician to develop extremely powerful models quickly and reliably. Results illustrate clearly what impacts a given target and how predictive and reliable the model is.

The tool is so simple to deploy that the previously employed distinction between operational and analytical CRM, which was needed when such diverse skills were required, can now be forgotten, and businesses can get on with building effective CRM solutions. The necessary analysis can be embedded in the day-to-day operational systems for the business domain experts to use as and when they require it without the need for outside assistance.

The Future

It should be remembered that business exists in order to make a profit. As the ecosystems that surround all enterprise become more complex, it becomes essential for all organizations to become more intelligent.

Analytics has been one of those areas that everyone agrees is important and should be given a high priority, but because of the issues surrounding costs, skills, difficulty in understanding the output and other well documented and universally understood reasons, it has never really been central to most business strategies. The challenges that business in the western economies face today make it essential that rapid and comprehensive transformation into customer-centric and intelligent adaptive organizations occur.

New tools now exist that are based on SRM and unique in their ability to give the power of advanced data mining to the business user. Traditional modeling tools can and do build very good models, but they come from a paradigm in which analytics are handled by professionals who do their work on behalf of the business user.

New approaches that enable predictive analytics to be embedded into the business process are the right way to go and are the future. Marketing managers and professional analysts should recognize this, move now to embrace the new tools alongside their existing technologies and encourage their business to embed true insight into their day-to-day processes.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access