Streaming data in the enterprise hits a tipping point
Data is transforming every industry, from financial services to retail to healthcare to transportation. The ability to react to data in the moment and engage in ongoing “conversations” with customers is what separates many winners and losers today.
Streaming data is the DNA of large-scale tech companies like Facebook, Google, Netflix and Uber that continuously redefine and improve businesses operations, how they engage with customers and how they out-maneuver the competition. The stream processing ecosystem has matured and is growing fast, and 2018 will be a watershed year for streaming in the enterprise.
Here are five predictions for what we can expect in 2018:
Stream processing technologies will become mainstream in the enterprise by the end of 2018, moving beyond technology companies.
At data Artisans, we are seeing strong adoption from large organizations in financial services, telecommunications, manufacturing and other industries. The adoption is accelerating as well and surpassing our expectations. Backing this up are analyst predictions that the streaming data applications market will reach more than $13 billion by 2021. (Streaming Analytics by Market Type Report, MarketsandMarkets, July 2016.)
Enterprises will invest in new products and tools to productionize and institutionalize data stream processing.
As companies are moving real-time data processing to large scale both in terms of data processed and number of applications, they will need seek out new tools that make it easy to run streaming applications production and reduce the manpower, cost and effort required.
Stream processing will expand beyond fast movement of data or simple analytical applications to operational applications that make true use of stateful capabilities.
Large global organizations across industries are adopting streaming data applications for fraud detection, sales and marketing management, predictive asset maintenance, real-time inventory, risk management, and operations management, among other use cases.
Evidence of this growth is the increase in Flink deployments outside a Hadoop context or even big data context (e.g., deployed on Kubernetes and managed by product teams rather than the Hadoop team). Apache Flink is also becoming more developer-friendly and can now be used without Hadoop, further opening up streaming data applications for developers who are not using Hadoop. Flink programs that do not rely on Hadoop components can now be much smaller, a benefit particularly in a container-based setup resulting in less network traffic and better performance.
The days in which we distinguish between “batch” and “stream” data processing will soon come to an end.
There is no fundamental reason for this distinction and the evolution of technology will make it disappear. Many applications need both of these capabilities so in the end we will talk about data processing.
Companies will realize business ROI faster with real-time data technologies than with Hadoop.
In our experience, the adoption of Flink in the enterprise starts with very specific use cases instead of open-ended projects which was very often the case with Hadoop. Companies realize business ROI faster because they get live applications up and running as the first step. For example, after one year of Flink being in production at Alibaba, Alibaba reported a 30 percent increase in conversion rate during Singles Day 2016 ($25 billion of merchandise sold in a single day this year).
2018 promises to be an exciting year with lots in store for streaming data practices.