Big data is six year old news. The Internet of Things data is bigger. It’s the hot new trend. It’s a flood of data. Billions of sensors never sleep, never stop talking. They send more data in an hour than millions of people in a day.
Terrified yet? Don’t be. No corporation wants terabytes flooding their network. It’s mostly noise that gets deleted anyway. Most IoT data is repetitive. And it goes stale in a few days. So filtering it, deduplicating, and compressing often shrinks terabytes to gigabytes.
Disk drive sales guys won’t like that. But most sensor data is low value or no value data. It should never leave the edge gateway servers that manage the sensors. Edge servers should filter, aggregate, compress, or discard boring data. Stuffing huge amounts of IoT data into the cloud is what pioneers are doing. Not the cognoscenti.
Advanced IoT architects tackle data volume at the source (keep your eye on Cisco). Not every sensor application must clog networks and saturate their data lake. Many will aggregate temporal data. Then they send it straight to the data warehouse for the big ROI payoffs.
This is a key lesson for us all in 2016. But it’s not a fait accompli – we have some tough programming ahead. It takes considerable effort to eliminate boring data without actually losing any information. Many IoT compression algorithms still need to be invented. Not all data will succumb. But most sensor data will.
That requires skilled analytic experts in 2016. Demand for data scientists, power users, and architects will soar even higher in 2016. Corporations will compete for these experts but will not fill their gaps.
Two trends are driving this: big data hype and board room discussions about algorithms.
Yes, visionary CEOs are debating algorithms for new business models and decision automation. Thank Uber and Tesla for alerting CEOs to take action. But this also means losing one analytic expert undermines dozens of projects for months.
The loss of top data analysts is driving corporations to collaborative analytic tools to retain tribal knowledge. New hires should not spend half their time backtracking, guessing, and searching for the right data. Veteran analysts should not spend hours and days tutoring recent hires.
If only there was a way to hold onto the exiting analytic expert's knowledge. This is good news for software vendors such as Alation, ProFinda, and Panorama. They provide tools that collect employee knowledge and foster collaboration on your analytics usage. They even analyze your analytics.
But software only helps if management makes it a priority to capture tribal knowledge. Try hollering “collaborate” once a month and “rush this task” every day. Employees will collaborate once a month.
Management discipline is not optional. In 2016, corporations that compete using analytics will invest in collaborative ideation tools and processes. Laggards will continue harassing HR for faster hiring. Which brings us to decision automation algorithms. It’s another way to solve the hiring pain. We watched hedge fund traders do this, never imagining it would permeate our own jobs.
Imagine alerting cashiers to staff up checkout lanes when many shoppers enter the frozen food aisle. Yes, everyone buys the ice cream last. Compare checkout queues to frozen food traffic and --presto– an automated management decision is made.
Google’s self-driving cars show algorithms can do tasks we thought only people could do. My 85-year-old mother needs a Google car. Algorithms are also making personalized purchase recommendations better than sales associates too. “People who bought this also bought these items.”
This gives rise to an analytic apps marketplace. Algorithms will be for sale in applications, small apps, and raw form in 2016. Think of it as iPhone style apps for analytics.
It’s an emerging market. That means app buyers are business managers, not IT. The more advanced apps will automate entire business process steps. This means we won’t be hiring 200 people offshore to watch business intelligence dashboards. They aren’t always good at it anyway -- too many missed trends and false positives.
New algorithms will see what people cannot. And they never miss anything, never take a coffee break. Algorithms will detect anomalies and emerging trends long before the CFO needs to know.
Next comes all the robots and drones driven by – you guessed it – algorithms. In 2016, algorithm stores will emerge on the internet. Buyers and SaaS renters will be in every corporation. In 2016, digital automation leaps to new heights until it becomes commonplace in 2017.
2016 is the year of even more awesome analytics. Every one of us will be disrupted. I want a self-driving car --now.
(About the author: Dan Graham is general manager at Enterprise Systems)