Before getting started, let’s take a quick multiple choice test. Which of the following is the best premise to begin a big data initiative?
- A. We have always generated information through the data warehouse and will continue to do it that way because those are the skills we have.
- B. If we don’t let IT set the agenda for information project, they will filibuster our efforts with security and architectural concerns -- so we will let them set the agenda and pace for data projects this year.
- C. I can only access what I have control over so I’ll limit my analyses to that.
- D. We need to deliver the right data at the right time to folks making business decisions to deliver the optimal services/products to our customers.
If you are resigned to A, B or C the rest of this article is probably not worth your time. But if you think D is the right answer, then you may face an arduous road ahead, but the many rewards at the end will make it more than worth it.
While everyone gives lip service to answer D, the fact is that there are many political obstacles to realizing this goal. First, you may have to upend a decades-old method of organizing and delivering data that is data warehouse-centric. You’ll be transferring a lot of the responsibility and control of information over to the business. And there will be deep fundamental changes in the organizational structure and process around delivering insights. For some, these steps are like walking through a political minefield.
New Way to Generate Insights
There is always room for improvement in any discipline, and there is plenty of room these days in the big data world as so many new opportunities and challenges battle for businesses’ attention. Insight generation, including core data management and delivery, must radically change in order to compete in this modern era.
Google, Facebook, and Yahoo developed and use new approaches and technologies to capitalize on their data assets. They had to, to compete. Still, big data advocates face some deeply entrenched opponents that look for any reason not to budge. They seem to start with the premise that the earth is the center of the universe and continue to expand their understanding from there. And no, that is not exaggerating the extent of the paradigm shift we are taking about.
Special Interest Hurdles
So -- if change is key to realizing value, and change is necessary, why are organizations giving in to internal special interests with agendas that are not aligned with the ultimate goal of delivering the right data, at the right time, to the right people? More importantly, what are executives caving to and what effect is that having on the success of big data initiatives?
Internal special interests block way too many initiatives and dull the edge of new, sometimes breakthrough insights. They are too invested in decades-old methods of organizing and delivering data. They cling to the responsibility and control of information; handing it over to the business feels like career suicide. And they freeze up at the idea of deep fundamental changes in their organizational structure and process around delivering insights. But they have more to fear from inaction that taking these steps forward.
What’s the solution?
Be Fast Off the Line
While there could be pages and pages written here about the technological advances and new data creation and organizational approaches, the key to successfully navigating this politically charged environment is to start small and low cost and deliver answers quickly. Something that fits in the quarterly budget for your department.
Services like Amazon AWS and Azure, not to mention a host of big data-specific players can spin up an environment capable of handling hundreds of terabytes in a half an hour. Data can be loaded and analyzed in a matter of a couple of weeks with best in class security and the environment can be shut down once the insights are generated and verified. This dynamism -- fitting the environment, effort, duration, and cost to the need -- shakes the idea that all data projects require months of architecting at the enterprise level to its core.
With this type of approach you can side-step the plethora of meetings trying to convince established players that they need to change. This is an analytics engagement, pure and simple, with a byproduct of having an industrial strength big data environment behind it. That environment can be around for a couple weeks or a couple years depending on the need.
Get Ready to Impress
Answers can now be delivered in time frames unimaginable two years ago. I do a lot of work in the pharmaceutical industry and it is possible to take three or four syndicated patient level data sets with billions of rows and have that data available in an analytic ready form in the cloud in three to five days.
Standard tools can access the data (SAS, R, common BI tools, Python, SQL) and internal analysts have everything they need to generate the best possible answer. And if questions come up, which they always do, the analysts can go back to the well anytime they want. This approach is very helpful to the acquisition team in evaluating potential drugs they want to go after to complement their existing line. The answers need to be delivered quickly and the nature of the data and questions asked are very ad-hoc, changing from month to month.
So what if the business started buying dynamic analytic ready data for their analysts versus static data sets that cannot be changed or queried further once delivered? What if these engagements cost the same as what you pay to access the enterprise data store now and deliver answers in weeks? Talk is cheap and results speak very loudly.
It has been my experience that once the process is proven this way, many internal detractors miraculously become advocates, helping big data applications and support get moving in earnest. Dynamic, short-duration environments and analytics provide a new tool in the business’s arsenal that can be sized and delivered on demand and can help get the big data ball rolling – right over those internal hurdles that once seemed so daunting.
(About the author: Rich Sokolosky is partner and practice leader for Life Sciences Data Strategy, Big Data, Business Intelligence, and Analytics for NewVantage Partners, a management consulting and managed services solutions firm).