March 4, 2011 – Dirty data, explaining data mining to others, and difficult access to information are the top challenges data miners face, according to a new study from Rexer Analytics.

In this fourth annual data mining study, Rexer Analytics asked 735 data miners from 60 countries 50 questions about their jobs, tools and challenges.

Only 8 percent rate their data quality as “very strong.” Only 13 percent of data miners rate their company’s analytic capabilities as “excellent.”

Rexer’s study gives an overall picture of the work methods of data miners. Data miners surveyed work primarily in CRM/marketing, the number one field for the past four years. Thus, customer understanding and retention are primary goals identified by respondents. Academia and financial services were the next two areas of concentration, but miners work in diverse fields, from retail to government.

Decision trees, regression and cluster analysis continue to form a triad of core algorithms for most, the study noted. Of the respondents, a third uses text mining, and another third plans to in the future.

About one-third of data miners typically build final models with 10 or fewer variables, while about 28 percent generally construct models with more than 45 variables.

Data mining most often occurs on a desktop or laptop computer, and frequently the data is stored locally. Model scoring typically happens using the same software used to develop models.

Data miners shared best practices for overcoming these challenges. “Use anomaly detection to flag records to put before subject matter experts,” is one suggestion to combat dirty data from the report. 

Data miners in Rexer’s survey are optimistic about growth, with 73 percent predicting increases in the number of projects. Growth in data mining adoption is the number one future trend identified.

More of our coverage on data mining can be read here.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access