The Fall of Intuition-Based Decisions and Rise of Little Data

Register now

Two dynamics are poised to transform the way companies operate: the exponential increase in data and the realization on the part of executives that predictive models and algorithms produce better decision guidance than human intuition alone.

While most managers agree on the importance of using data, many believe that the big data hype often associated with companies like Google and Amazon doesn’t apply to them. Or perhaps they are intimidated with the internal resources and hefty investment required to tap into that data. Others may be skeptical that the use of predictive models can actually lead to better business performance.

As a result, managers too often fall back on subjective, intuition-based methods to make business decisions, missing the benefits reaped by those who have tapped into the data available to them. The reality is that companies have more data than they think, they need less data than they think, and more than 50 years of research supports the notion that predictive models consistently outperform human decision-making abilities.

Big Data versus Little Data

When big data is described, it’s usually in terms of the three Vs: volume, velocity and variety. But the benefit of the big data movement has less to do with the volume of data and more to do with driving action and value out of data by applying algorithms and predictive models to solve specific business problems. To that end, companies don’t need petabytes or even terabytes of data similar to the magnitude of data collected by Google, Facebook and Amazon.

Most B2B companies are only scratching the surface of the business benefit hiding in the data they already have. When B2B companies think big data, they often imagine a multi-million-dollar investment and a resource-intensive IT project lasting a year or more. Many B2B companies are not ready for those risky and expensive projects, especially when executives are unconvinced of the benefits.

The good news is that all B2B companies can significantly improve business performance through the proper use of their existing data (or little data). Some forward-thinking B2B companies already realize that they don’t necessarily need to look outside the data housed in their data warehouse, ERP or CRM systems to make significant strides toward better decisions.

The Benefits of Little Data

Internal company data, specifically transaction data that companies collect in the normal course of doing business, can provide managers and front-line sales and marketing employees with a tremendous amount of actionable insight. Consider these examples:

1. Growth through Cross-Selling. For many established B2B companies with aggressive revenue targets, sales teams spend the majority of their time trying to maintain and grow existing customer accounts. The challenge is determining which additional products to sell to which customers and which opportunities to prioritize. Sales teams know there are products that their customers are currently buying from competitors that they could be buying from them, but they don’t know where those opportunities exist. This becomes especially challenging for sales reps with large books of business. However, purchase patterns, identified by applying algorithms to companies’ existing data, can reveal which additional products each customer should be buying, helping sales reps prioritize the best opportunities for wallet-share expansion.

2. Preempting Customer Defection and Churn. In most situations, retaining a customer costs less than acquiring a new one. For B2B companies, the loss of one valuable customer is extremely expensive in terms of future revenue and profit. And once that customer is gone, getting that it back can be challenging and expensive. If your sales reps have large books of business, it’s difficult for them to stay on top of the small changes in behavior that are early indicators of churn or defection. As a result, customer churn is often only recognized when it becomes very obvious, and by that time, it may be too late. Fortunately, predictive analysis of companies’ existing transaction data can show the early signs of defection and provide sales teams with the opportunity to preempt defection.

3. Better Pricing Decisions. There are far too many manufacturers and distributors still using “cost-plus” and/or “list minus” rules-of-thumb, or outright guesswork, to guide their day-to-day pricing decisions. As a result, there’s typically a lot of room for improvement in this area, and most companies have barely scratched the surface of what’s really possible when it comes to profitable pricing in their marketplace. Through advanced price optimization, a company can identify all of the unique pricing segments that exist in its marketplace. It can increase the number of addressable segments in its business by orders of magnitude and maximize profit potential. With scientific precision, companies can finally develop a pricing model that reflects the true level of complexity and granularity that exists in their particular marketplace.

4. Recognizing Sales Upticks. When companies receive an unexpected sales uptick from a customer, sales reps should not only follow up to say thank you, but they should also investigate if the customer has a need for additional complementary products. However, when sales reps are managing thousands of customer accounts at once, the challenge is recognizing when a sales uptick takes place. Fortunately, with analysis of their existing data, companies can easily track sales upticks and up-sell opportunities that will offer a competitive advantage and help grow wallet share.

5. New Product Introduction. Companies often introduce multiple versions of a product line to capture different customer segments. One challenge, and inherent risk of versioning, is the cannibalization of premium lines by lower margin, “fighter” products. Knowing which customers to target — those who are not buying or have been slowly defecting from your higher-end product — can help you grow revenue and wallet share with the new product while protecting your existing business. By identifying customer opportunities for individual products through existing transaction and customer data, companies can make more strategic decisions about new product launches and base their decisions on quantifiable market demand and revenue opportunity.

Ultimately, the insight gained from companies’ existing transaction and customer data is critical to strong financial growth. Most importantly, the data needs to be processed and presented in a way that is actionable for front-line employees.

The Fall of Intuition-Based Decisions

Historically, managers have relied on experience and intuition to make business decisions. Many B2B companies, however, have grown to a point where there are simply too many decisions to be made and not enough time or information to make the best possible decision each and every time. The availability of data and predictive models is changing that, as it is now possible to significantly improve decisions at all levels in an organization.

It’s not typically the technology, cost or analytical skill gaps that prevent organizations from embracing predictive models. It is primarily because key executives still believe that experience and judgment are sufficient in making a good decision.

However, companies no longer have to settle for “okay” decisions. In fact, companies that fail to use data and predictive models to make decisions are likely to be surpassed by their competitors. That failure is less likely to result from technology challenges; rather, it’s more likely to be caused by managers’ resistance to trust data and mathematical models to guide their decisions.

As a result, too many companies leave important decisions about which customers to call on, which products to sell, and what prices to quote up to the best judgment of front-line sales and marketing employees. Attempts to provide access to data and reports often prove futile, as front-line employees generally crave direction and specific, actionable guidance. The end result is millions of decisions that contain slight errors. Those errors have a big cumulative impact, potentially adding up to millions of dollars in lost revenue and profit.

Even the most well-intentioned employees can’t possibly make the best decision each and every time. Humans are persistently inconsistent in their decision-making and predictions. Human reliance on memory of data and subjective inferences are to blame. In 1954, psychology professor Paul Meehl published the book “Clinical vs. Statistical Prediction: A Theoretical Analysis and a Review of the Evidence,” in which he made the claim that mechanical methods of data combination, such as using algorithms, always outperform clinical, or “subjective,” methods when making predictions.

The book caused significant controversy among psychologists, but time and again since then, it has been scientifically proven that simple algorithms make better predictions than humans. According to According to a paper published by Michael Bishop and J.D. Trout, called “50 Years of Successful Predictive Modeling Should Be Enough: Lessons for Philosophy of Science," predictive models have proven more reliable than people at predicting everything from the success of students’ academic performance to the presence, location and cause of brain damage. A model can take into account any type of data, including experts’ input or judgment, but once given, the model makes the prediction. In other words, expert intuition is a valid input in a model, just not the only input.

Despite more than 50 years of research and proof on the subject, most companies still make the vast majority of decisions by relying on experience and intuition or by simply viewing data and drawing conclusions. Predictive models are rarely used to guide decisions.

However, this is not a “man versus machine” story, but rather “man plus machine,” where the combined result is a better outcome. Managers should provide input into algorithms then deliver guidance directly to employees, rather than a dictated best guess. Information needs to be pushed directly to the employees in the field. Arming them with this level of guidance will result in decisions that are optimized, consistent and aligned with company strategy.

In the same way we now struggle to imagine how business worked before computers and the Internet were pervasive, years from now we will look back at subjective decision-making in disbelief. Smart companies recognize this and are moving fast to exploit their little data to enable better decision-making, resulting in a competitive advantage and better financial performance.

For reprint and licensing requests for this article, click here.