A growing number of organizations are implementing data lakes to better store, access, and analyze data. But the reality is that data lakes are hard to manage. At the recent Strata & Hadoop World conference in New York, Information Management spoke with Asaf Somekh, founder and CEO at iguazio, on why this is so hard.
Information Management: What are the most common themes that you heard from attendees?
Asaf Somekh: We noticed signs that the industry is maturing and were happy to see so many conference attendees who clearly were not just vendors and developers, but rather data scientists, architects and CIOs of enterprise organizations.
Having said that, their companies are clearly having problems with adoption of big data, and they were preoccupied with the over complexity and high cost of managing data lakes. Another theme we saw is related to security, as participants were trying to understand what levels of granularity would be possible for them to allocate various permissions of anonymize data.
IM: What are the most common data challenges that attendees are facing?
Somekh: Complexity, complexity, complexity… and even manual copying… due to too many data silos and long transformations between data models.
IM: What are the most surprising things that you are hearing from attendees?
Somekh: That the majority of enterprises still believe they need to store all data on premises in order to guarantee security and governance.
Q: What does your company view as the top data issues or challenges in 2016?
Somekh: The biggest challenge in our view is that of secure data sharing, which is also differential. What we mean here is a most basic inability to easily share anonymized data in real-time. A common example is that of two international teams of traders from the same bank which are still unable to effectively benefit from each other’s data.
The second challenge we see is that of ETLs slowing everything down. The result is that fraud detection teams are in reality only fraud reporting teams, as they are usually unable to prevent fraud in real-time.
The bottom line in terms of challenges is simplicity, performance, security and agility. Most enterprise organizations have yet to benefit from them.
IM: How do these themes and challenges relate to your company’s market strategy this year?
Somekh: iguazio addresses these challenges directly by delivering a fully integrated and elastically scalable cloud-native service. We built our solution from the ground up to maximize CPU utilization and leverage the benefits of non-volatile memory, flash and dense storage. iguazio redefines the category with enterprise-grade capabilities and an application-centric, role-based self-service operation.