Slideshow 7 top challenges to working with data

  • June 21 2018, 5:31am EDT
8 Images Total

Overwhelmed by data

Data professionals may often feel that they are drowning in data, making it difficult to maintain consistency, identify 'good' data, or to derive valuable insights from it. "The Definitive Data Operations Report" from data operations platform provider Nexla, looks at the top challenges that data professionals say they face in managing it all.

39 percent say data format consistency

Given that data pros spend 17 percent of their time on data cleaning, it should come as no surprise that it tops the list of challenges they face. Thirty nine percent of respondents said data format consistency is a challenge for them.

Content Continues Below

36 percent say data integration

Almost all data pros report that their company is working with artificial and machine learning, making data integration all the more important. This is up significantly from 2017, when ‘only’ 70 percent of respondents reported that their companies were working on ML or AI.

35 percent say reliability of data pipelines

Of those companies that currently share data with third parties, 48 percent say they share with ten or more partners. Thirty seven percent of these companies send their data in real-time, and 33 percent send the data daily.

32 percent say access to external data

Companies are increasingly relying on data from outside. In 2018, 77 percent of respondents said their company currently ingests data from third parties. This is up from 60 percent last year. Thirty two percent cite access to external data as a challenge, suggesting inter-company data remains a challenge.

Content Continues Below

32 percent say data science / analytical skillset

Not surprisingly, the majority of respondents said their companies have plans to hire DataOps professionals in the next 12 months. When looking at the 73 percent of respondents who said they are planning to hire, two-thirds reported they did not think there were enough backend resources available.

26 percent say governance / compliance

Issues related to data governance and compliance have risen in recent months, driven in part by new data management and data privacy regulations such as the General Data Protection Regulation (GDPR), which places tough new standards on how personal data is held. But data governance is also impacted by the growing volumes of data being collected by a greater number of devices and the Internet of Things.

19 percent say access to internal data

Interestingly, only 19 percent of respondents cite challenges with internal data access, suggesting that efforts to break down data silos have often been successful.