Will AI and machine learning replace the data scientist?

Register now

The role of a data scientist ranks number one in Glassdoor’s annual survey of the year’s best jobs? This marks the second year in a row that the data scientist is at the top of that list. With the amount of data growing exponentially every day these professionals are needed to sift through and find insights that will help organizations be more efficient and effective with customers, suppliers, and partners.

At the same time, new technologies that once seemed far off, are now poised to automate human analysis and decision making. It’s been six years since IBM’s Watson beat former Jeopardy! champions to win the $1 million prize. Since 2013, Watson has been making utilization management decisions at Memorial Sloan Kettering Cancer Center where IBM say that 90% of nurses in the field who are using Watson are following its guidance.

The Data Scientists vs. Artificial Intelligence

We are now seeing machine learning and artificial intelligence (AI) being used in other everyday applications, including Apple’s Siri, Amazon’s Alexa, and Tesla’s self-driving cars. These applications are making decisions and taking actions that humans have always done in the past. In fact, Gartner says that by 2020, 85% of customer interactions will be managed without a human.

AI applications comb through loads of data to create models capable of predicting and sharing outcomes. It is no longer programmed manually, but is capable of learning and continuously improving on its own. Scientific American says that in the next 10 to 20 years, nearly half of today’s jobs will be at risk due to AI algorithms. So, you might very well ask, will AI and machine learning eventually replace the data scientist? It’s a fair question and maybe we should ask Siri.

All kidding aside, I don’t think the data scientist is going away anytime soon. The data scientist will always be needed to coordinate new business needs with AI. Their job is, and will be for the foreseeable future, to translate the business problem into data sets and algorithms, testing them, and then setting them in motion. Every business problem requires its own unique solution that only the data scientist can solve. Unfortunately, you can’t take the code in healthcare for predicting heart attacks and reuse it in auto insurance to determine accidents.

The Opportunity

Companies today are wrestling with data varieties and volumes they’ve never previously seen. Data is being produced from every machine and process. It’s creating a myriad of data types, including structured and un-structured, qualitative, and quantitative, continuous and discrete, nominal and ordinal. It is high volume, high velocity, and high variety. Data scientists are just the ones to understand and determine the data requirements needed for the optimal AI and machine learning solution.

What we will see in the near future is more and more technology to support the work of the data scientist. New types of data will be captured and made available. Data mining and analytical tools will get easier to use. More data analysts will become business analysts, using the data sets produced by the data scientists. And, most importantly, it will get easier to integrate the artificial intelligence and machine learning into production applications, helping to automate decision making. This is where organizations will realize the true value of big data.

Arming the Data Scientist and the Organization

The thing to remember is that data scientists cannot do their job in a vacuum. They need to be armed with the appropriate tools and data in order to tell the most accurate story. First, that means giving them access to a variety of data sources that represent how business gets done. These should include both internal systems and external systems in order to provide context and key pieces in the data puzzle.

Secondly, we need new tools that help the data scientist extract, merge, reduce, analyze, and present the data. These tools must be integrated in order to make this process more accurate and more efficient. The insights found by the data scientist then need to be operationalized so that they can be tracked going forward, looking for changes and trends. This is where the data scientist’s work really becomes integrated into the daily operations, allowing the business to learn and collaborate together.

Start Your Search Sooner Rather than Later

On top of being the number one job, the data scientist has also been called the “sexiest job of the 21st century.” With honors like that, you would think there would be a flood of applicants. Truth is, by just next year, the number of data science positions will exceed 490,000, but there will be less than 200,000 data scientists to fill these positions, according to a McKinsey study.

A good data scientist is truly a fortunate find. A business-savvy person with the advanced technical skills to find insightful nuggets in mountains of data can drive your business forward in ways you may not yet have imagined. After all, it was the data scientist who created personal recommendations on websites like Amazon and Netflix, created the LinkedIn network, and custom ads and content on sites like Facebook, generating millions of dollars for these companies.

For reprint and licensing requests for this article, click here.