Capital One, the Federal Reserve, Google, New York City and the U.S. Army all have at least one thing in common: they each employ a chief data officer to oversee their big data programs.

The emergence of the Chief Data Officer closely mirrors a trend we saw in the 1980s when companies began to adopt personal computers, servers and digital technology en masse. At the time, some organizations realized that such a massive initiative required a combination of expertise and executive-level leadership. As a result, the chief information officer was born, and, by the 1990s, the CIO was ubiquitous on executive teams.

We believe that CDOs will become universal as well. Companies in retail, manufacturing, agriculture, resource extraction, finance and professional services and tech, among others, will soon realize that big data is too specialized and too much of a responsibility for the CIO or CTO. As we will illustrate, big data initiatives are high value but also high risk endeavors. A data scientist with executive powers is critical to making sure data science delivers. 

What is the Opportunity?

The emergence of big data and CDOs is fairly easy to explain: There is a ton of data and organizations want people who can monetize it.  

International Data Corporation (IDC) reports that the digital universe will grow 300-fold between 2005 and 2020 to a total of 40 trillion gigabytes. Yet, according to IDC, only one percent of this data is currently being analyzed. That gap is the opportunity. Companies capable of gathering, structuring, analyzing and using that data stand to gain a competitive edge in their respective marketplaces.

Many companies prefer to hide their big data success because they don’t want competitors to know what they’re up to. But two recent examples should give you an idea of what a CDO and data science team can accomplish.

In April 2013, a business professor, Tobias Preis, and two physicists published a paper titled “Quantifying Trading Behavior in Financial Markets Using Google Trends.” They found that as the volume of searches for words like “debt,” “portfolio” and “stocks” fell, the Dow Jones tended to rise. Indeed, a stock trader using what the authors term “a Google Trends strategy” based purely on search volumes of “debt” would have yielded a 326 percent return between 2004 and 2011. By comparison, very high performing mutual funds get a return of between 10 and 16 percent. Unsurprisingly, the BBC reports that the three researchers were approached by executives in the financial industry.


Another great example is Gilt Group, which reached $500 million in sales in just five years. As a report from McKinsey & Co. describes, Gilt Groupe mines extensive customer data to personalize all e-communications. Consumers on Gilt Group’s email list, for instance, will receive one of 3,000 different messages depending on their expressed interests. The messaging system is a joint project between company data scientists, who built the algorithm, and creative teams, which created imagery and language that appeal to each segment of customers. The result is an intelligent, personal invitation to shop for goods that the customer is likely to find appealing.

Essentially, organizations realize that they can collect enormous volumes of data that provide opportunities to improve nearly every dimension of their business. They just need people who can make it happen.


The Data Scientists

To understand the need for CDOs, it also helps to understand the role data scientists play in organizations.

Today, the best data scientists come from companies with existing data programs: Amazon, Google, LinkedIn, Microsoft, Netflix, Twitter and Yahoo! to name a few. They are mathematicians, computer scientists, academically trained scientists (astrophysicists, ecologists, biologists, etc.), hackers and software engineers who use the scientific method. They ask a question, develop a hypothesis, collect data (often through an experiment), analyze the results, report their findings and then often engineer them into applications.

Data scientists are intensely curious. They do not ask, “How many people clicked this button?” That is just one piece of raw, unstructured data. Instead, they might ask, “At what discount percentage is a white male, 18- to 25-year-old shopper most likely to add an item presented at the initial checkout screen?” Then, the data scientists can determine which items to present, where to locate the offer button on the screen and ultimately create revenue.

Thus, data scientists often show other departments the unintended negative consequences of their actions, or what they could be doing better. The best data scientists develop algorithms and models that can be used to power recommendation engines, inventory management systems and other applications. In a sense, they are a hybrid between R&D and strategy consultants who leverages an organization’s full array of data instruments and test environments to create value.

Like R&D and consulting, they also require a heavy technological and financial commitment. According to Wikibon, Google spent over $3 billion on big data infrastructure in 2012 and Facebook spent $1 billion. For any company, this is a lot to spend on experimentation.

How CDOs Make Data Science Pay Off

Data science, as described above, is time-consuming, expensive and risky. It involves failures and it often creates tension between the data scientists and departments that are asked to change their ways or implement data-driven applications. To minimize the risks and maximize the results, companies (and government entities) want a CDO with the skills and influence to make sure data science pays off.

A CDO can create a data-driven culture where information is gathered ethically, shared widely and used to grow revenue and cut costs. Ideally, a CDO oversees a team that handles all data analysis — from social media, marketing and advertising to pricing, customer service and operational processes. CDOs and their teams, unbound by departmental divisions, can become a fountainhead of intelligence and solutions that aim to make every department more effective.

But because data scientists may operate across departments and frequently challenge the status quo, they often butt heads with company culture and co-workers.  For example, Greg Linden, formerly a data scientist at Amazon, wanted to make shopping recommendations based on items already in a shopper’s cart. As Linden wrote in his blog in 2006, “we had an opportunity to personalize impulse buys.” He coded a prototype and modified a test site of the shopping cart page so that it would offer recommendations based on his algorithm.

However, a senior VP of marketing said it would distract shoppers from checking out and forbade Linden from working on the project. Fortunately, he ignored the VP and others who tried to make a case against testing the prototype. In tests, the shopping cart recommender was a tremendous success — in fact, Amazon realized that not having such a recommendation engine was costing the company money.

In the absence of a CDO, someone like Linden has to risk his job just to test a hypothesis. When a CDO makes it clear that such experimentation is expected and enrolls fellow executives in the opportunities, the CDO can free data scientists to investigate issues they might otherwise be afraid or unable to investigate. Ultimately, a CDO allows data scientists to operate free from the constraints of company politics and HiPPOs (Highly Paid Person’s Opinions).

Still, a CDO’s responsibility is not to be right all the time or to make other people look bad — the CDO’s job is to test the assumptions, procedures and beliefs that, if verified or challenged, would have the greatest effect on short-term profit and long-term health, and sometimes even the trade-off between the two.

Given the high costs and risks of data science, the lengthy project cycles and necessity for some failures, organizations will demand the oversight, strategic direction and accountability that a chief data officer can uniquely provide.

The role of big data is simply too big for the plate of a chief information officer or chief technology officer. While these executives will surely partner with the CDO, most CIOs and CTOs do not possess the knowledge and experience to guide a big data program, and most do not have the bandwidth to cultivate and champion a data-driven culture.

By 2018, McKinsey suggests that the U.S. alone “could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.”

As universities begin to create data science programs and as companies with big data programs continue to train and develop data scientists, the pool of candidates will grow, but not nearly as fast as demand.

This means that the best CDOs and data science teams will be both a source of value and a beacon for future talent. The data science leadership that can offer the best training, most resources and highest impact projects will in the long run be able to attract data scientists that other companies cannot.

In our opinion, companies that can find a qualified CDO and build a data science program stand to become more competitive, innovative and nimble. Indeed, their data science teams will be more than able and willing to measure their value to the company.

Adam Charlson is an executive vice president and managing director of the West Coast for DHR International, as well as a member of the firm's technology practice. Over the course of his career, he has recruited senior leaders for such organizations as Adobe Systems, Convergys, Experian, FICO, Google, HP, Infor Global Solutions, Informatica, JDA Software, Microsoft, PayPal, Progress Software,, SAP, Sitel, and TeleTech, to name a few.

Kristen Barge is a principal in the DHR International San Francisco office and is responsible for conducting senior level retained executive search assignments. Her primary industry focus is technology, digital and media marketplaces.