Adoption of, satisfaction with, big data on the rise
A growing number of organizations are investing in big data deployments, and more of them are reporting success with those efforts.
Those are among the findings of a new study by Progress Software that looked at the rate of adoption of big data projects, and how organizations are typically rating their experiences with big data. According to Sumit Sarkar, chief data evangelist at Progress, more than 1,200 business and information technology professionals were surveyed for the study, the “2017 Progress Data Connectivity Outlook.”
Information Management spoke with Sarkar about the study findings, including how data security issues and the pending General Data Protection Regulation will impact big data strategies.
Information Management: Your research shows an increase in big data adoption among organizations from 50 percent to 61 percent. How do organizations typically rate their big data deployments?
Sumit Sarkar: Based on the demographics of our audience, positive responses signal a mature deployment that is proving business value since they’re investing in commercial data connectivity for enhanced performance and enterprise security. This was not the case a number of years ago when the majority of organizations were kicking the tires on Hadoop, and not yet ready to incorporate it into the enterprise wide data infrastructure.
IM: The number one reported challenge is dealing with the growing number of data sources that organizations must manage. Specifically how does this make things more challenging, and what are organizations trying to do to improve that?
Sarkar: The majority of data teams, comprised of data engineers, data scientists and data analysts, are highly skilled. However their ability to produce business insights is based on access to data. In addition to more data sources, we’re seeing changes in the surface area of APIs, limited access to databases and an increasing number of firewall boundaries.
The inability to access this data limits the success of the entire investment in data analytics, and organizations are looking to adopting interoperable industry standards for SQL (ODBC, JDBC) and REST (OData) to provide data teams frictionless data access.
IM: How does the growing number of these data sources impact the data security efforts of organizations?
Sarkar: Each new data source introduces additional complexity with encryption levels, authentication methods and workflows. In addition, many organizations need to factor in identity management and data lineage to track data access for regulators across different industries and government entities.
Then there is the threat of security breaches making organizations increasingly risk averse which makes it challenging to adopt the latest disruptive technology on day one which is represented in the survey as technologies consolidate and get enterprise ready over time.”
IM: You cite that 66 percent of respondents must comply with data protection laws. Do you have any sense of how the GDPR will impact organizations in this regard?
Sarkar: We’re seeing a large number of data ingestion from SaaS applications that store personal data that is subject to GDPR. In fact, the largest amount of source data being accessed in our DataDirect Cloud service is from sales and marketing SaaS applications, and their adoption ranks highly in the survey responses. We expect organizations will need to start monitoring the user roles and API usage from these SaaS applications to ensure compliance.