Steps to make your first big data initiative a success
Big data analytics offers revolutionary capabilities but achieving results requires a strategic vision—and a view into how it can create business value.
Over the last few years, the term "big data" has evolved from a vague concept into a mainstream business strategy. But somewhere at the intersection of conceptual promise and real-world implementation, many business leaders, including CIOs, have faced a tough realization: putting the concept to work requires different tools, technologies and strategies than in the past.
Imagine you are a CIO who has identified a business use case for big data analytics but is struggling with how to go about executing it. Tools, platforms, skillsets and domain expertise - all these will weigh on your mind as you try to put together a team.
Bringing in an external partner is a course that many organizations pursue, whether for consulting or for full implementation. The reason is simple: an experienced partner presumably knows best as to what works and where things can fail. So they can help limit your downside risk and at the same time maximize your upside gain.
These relationships are also learning experiences for vendors. For example, Valiance recently did a big data analytics project with an enterprise communications firm whereby we were supposed to extract customer insights from the terabytes of unstructured data they had. We were responsible for creating in house a big data infrastructure to store and process data, enable data mining, create machine learning algorithms and finally delivering the results through an API.
Here are some key lessons we learned from this engagement, and we recommend you to keep same in mind when engaging external help.
Know your partner’s strengths and weaknesses
It’s best to clearly understand technical approach used by your partner for data mining or machine learning along with related limitations to avoid disappointment later. There are limitations to what one can achieve with all the best algorithms available at your hand. You should definitely spend some time in understanding how related technologies and algorithms work. Understanding where outcomes may not be favorable can help you manage stakeholder expectations.
Exact infrastructure needs are hard to estimate
Big data analytics project demand infrastructure and often it’s difficult to exactly estimate capacity requirements in the start. Going for a fixed investment complicates the matter latter when your team needs additional capacity and that too variable. It’s best to consider using cloud infrastructure for your project if there can be uncertainties with processing workloads.
Defining rational measures of success
Big data promises to be revolutionary technology with answers to all your problems. That will not be the case. It’s advisable to define achievable and tangible measures of success rather than having irrational expectations. Big data isn’t the answer to all your problems.
Collaborate throughout the project, not just towards the end
Big data analytics project demand more collaboration and attention than typical IT projects. You need to understand the data processing cycle and related challenges. You will surely discover issues related to data quality. Participate in analytics discussions. You will understand how algorithms are used and may later help you in building your own teams.
Learn to accept the downside!
Even though your partner may have tried their best, outcomes may not come the way you hope or expect. There could be multiple reasons for this, starting with the data. Not even the best of algorithms or teams can help if your data doesn’t support the conclusions you need to reach. You partner probably couldn’t have predicted this at the start.