Success with data preparation often tied to business intelligence
Dresner Advisory Services has published the 2019 Data Preparation Market Study, which addresses perceptions and intentions surrounding data preparation, defined as a capacity for users to model, prepare and combine data prior to analysis.
The 5th edition of the report has been renamed from “end user data preparation” to simply “data preparation” to reflect a shift within the market that includes business professionals along with IT as key users of the technology. It examines end user requirements such as usability, integration and manipulation features, as well as output and deployment options. It also covers industry support for data preparation and compares user requirements to industry capabilities.
According to the 2019 report, data preparation is ranked 14th out of 33 business intelligence technologies and initiatives under study. Sixty three percent of all respondents indicate that data preparation is either “critical” or “very important”. Seventy-eight percent of respondents say their current data preparation approach is highly or very effective, with confidence increasing over time, according to the report. Success with data preparation correlates to success with business intelligence.
Information Management spoke with Research Director Jim Ericson, one of the authors of the report, on its key findings and what they mean for data professionals.
Information Management: What were the most significant findings from the most recent study?
Jim Ericson: Data preparation has proven to be a ‘sticky’ topic with high importance to users over time. Our audience tells us that success with data preparation correlates to success with business intelligence. Capable data preparation naturally leads to exploration, new insights, better sharing and confidence because of the ability to ask questions with fewer long cycles of technical support. Our audience tells us they’re getting better at it.
IM: How did this year’s finding differ from what prior studies have found?
Ericson: User requirements have been largely steady over time, and this is our fifth annual study. Basic manipulations like aggregate and pivot and the ability to combine data sources are still the nuts and bolts of data preparation. One difference is that this year, for the first time, cloud-based software and services now outnumber on-premise products, which gives more choice and entry points for users.
IM: What was the most surprising finding in your view?
Ericson: It may surprise some that the most important data preparation outputs are to flat files like Excel and CSV, and not to big data formats and services. This tells us something about the way users are grappling first and most often with in-house data and using data preparation to improve their day-to-day working processes. Third-party services from all the big names are slowly gaining traction and companies more often use third-party data, but the bulk of the daily lifting and reporting has not changed much for most organizations.
IM: Your study says that nearly two-thirds (63 percent) of respondents described data preparation as either critical or very important. How does that compare with what you would hope to see for results?
Ericson: I think it’s a powerful statement and that number gets close to 90 percent when people are asked if data preparation is, at minimum, ‘important.’ We define data preparation as the capability for a variety of users – both business and IT– to model, prepare, and combine data prior to analysis. Those numbers make for a pretty broad audience.
IM: The overall study results seem to present good news when it comes to how organizations view data preparation. How so and why so?
Ericson: Data preparation is an increasingly distributed process that leads to better analysis and decision-making. That the data preparation market is robust and well supported, and will continue to evolve with more bundled tools and service models is good news. There are always limits to automation and vendors will need to keep their eye on the ball as new data architectures and formats move to the mainstream. But when you consider the traditional complexity of data integration, cleansing and other tasks, data preparation has gotten its rightful attention and empowered more users.
IM: What do you consider the most important lessons from the study results?
Ericson: It’s important that organizations continue to review, observe and improve their data preparation capabilities and maybe raise expectations in a maturing market. There’s a lot of analyst expertise at work in organizations, but also opportunities to empower bigger audiences in ways that improve their daily work processes.