Today, a typical enterprise can access enormous amounts of data, and the huge volumes of data keep growing. The very high volume, velocity and variety of this data make the enterprise business data environment highly chaotic, complex and dynamic.

Navigating the tremendous quantities of data and driving visible business results can be a complex challenge. A survey of 3,000 business executives and analysts by MIT Sloan Management Review and IBM institute of Business Value found that the majority of organizations are still looking for better ways to obtain business value from their data.

Along with the huge amounts of data, there has been significant growth in the systems and technologies required to leverage data. The systems, processes and technologies for big data analytics are a very hot topic right now. We have heard some success stories and benefits of using big data analytics; however, we don’t hear much about the inherent challenges. Larry Bonfante, CIO of the United States Tennis Association and founder of CIO Bench Coach, LLC, described what he sees as the largest challenge with big data: “I have memories of my mother telling me, when I was a kid, not to bite off more than I could chew.” Similarly, Michael Wu, Lithium's Principal Scientist of Analytics, commented on the challenge of dealing with big data in more detail: “Most of the data are noise and only a tiny fraction is the signal.” It is not easy to distinctly identify and catch that tiny signal in the middle of all that noise.

Despite the challenges, big data is an emerging field that has created a lot of opportunities.  There are very powerful tools and technologies, with revolutionary functionality, that generate endless opportunities to create big impact and big value, if used effectively. For example, health care providers can use it to improve outcomes and cost structures. Pharmaceutical and biomedical companies can find new cures and analyze genetic information. Dating websites can use it to help their customers find the most compatible partner, and schools and educational institutions can help “at risk” students. Enterprises can leverage the same technology to generate new customer knowledge and insights. More specifically, enterprises can connect diverse data points, combine a wide range of data from multiple sources (such as customer behavior, psychographics and customer lifetime events) to create better products and more accurate profiles and predictions. However, the success of big data analytics in driving business value depends on five main steps.

1. Start with a Strategy

In the middle of all the hype, it is very important to be aware that adopting big data technology is not a magic bullet; it won’t provide instant agility, dynamic capability or resultant competitive success. The secret ingredient for building dynamic capability and competitive success is crafting the proper strategy to effectively leverage quickly evolving data and technologies. To effectively leverage the technologies, one needs to adopt it as a strategic weapon, not just an add-on technological tool. Amazon is a good example of the strategic use of big data. Amazon knew that low prices are key to its competitive advantage, and competing with off-site retail stores on the basis of price has been a crucial strategic challenge for the company. It leveraged big data technology to deal with the crucial strategic issue and offered customers up to five dollars off their purchases if they compare prices using their mobile phone application in a store. With this initiative, Amazon is gathering competitive intelligence on prices in stores and at the same time increasing use of its bar code scanning application.

Many companies are shying away from attacking the biggest strategic challenge head on like Amazon, preferring a more cautious approach of starting small to get “quick wins.” However, benefits of attacking strategic issues with big data analytics include involvement of top management due to the critical nature of the issue, higher attention and investment, and better talent and technology access, which are keys to driving business value and impact.

2. Know your Requirements

Before picking or adopting any kind of analytics technology, it is crucial to understand the actual data requirements. Sometimes, when a company begins with the clear strategy and questions in mind, it is already possible for them to access the right data (internally or through third-party data services), in which case an upgrade to the infrastructure to support big data may not even be needed. However, some strategic initiatives require you to own data, for use now and in the future. In that case, companies may need to utilize a new set of technologies for data capture, storage and analytics to make the big data accessible, relevant and manageable. To manage growing volumes of big data, it is crucial to create a fast, efficient and simple data integration environment. Despite the technological advancements, these tools and technologies are still new and not easily usable in an enterprise environment. Often, these tools require large technical teams; the hardest part is balancing the effectiveness of the technology with the capital and operational cost constraints. 

3. Access Talent

In the context of big data, analytic technologies have evolved faster than the workforce skills; these technologies have become relatively inexpensive, but it is not easy to find talented people who can make sense of it. It’s important to invest in skilled and talented people; just throwing money into a project or hiring a large number of unskilled people won’t necessarily solve the problems. There are a lot of examples about how analytics provides highly powerful insights in the right hands. However, when these powerful tools are put in the wrong hands, it can be wasteful or even disastrous. All your investments in big data analytics may lead to more rigidity and incompetence. Trying to search for meaningful stories in huge piles of data requires a relentless desire for exploration, constant experimentation, enormous skepticism and the proper assessment and feedback mechanisms.

4. Build Culture and Processes

Again, big data technologies have also evolved faster than the workplace processes can handle them. Companies need to build workflows, proper policies and a dynamic culture of experimentation to facilitate the use of big data. They also need to develop the right assessment and feedback mechanisms to increase the predictive and explanatory powers of the technology and explain counterintuitive findings. I believe the real impact of big data will be more visible when business executives are able to use these tools themselves to solve their problems, not just data scientists. There’s more opportunity for follow-up, in terms of actual programs and initiatives, when the business executives are able to experiment and play with the data.

5. Address Privacy and Governance issues

It is extremely important to be aware of the inherent risks of big data. It’s important to address security, privacy and governance issues. Within an organization, different groups need to clearly define data ownership and access to ensure that the proper protections are in place. Senior management needs to pay attention to increasing data security issues and cyber risks. Until now, consumers have traded lots of their personal data to fulfill other personal needs, and they expect companies to respect privacy and be proactively responsible for it, not to manipulate or misuse it. Consumers expect bigger and more technologically advanced companies to have clear and transparent privacy policies, setting an examples of ethical leadership.

Overall, powerful tools and technologies for big data analytics have offered opportunities for big impact and value. However, leveraging these opportunities is without its own hurdles. If used effectively, these tools and technology can create miracles; however, when put in the wrong hands, or used ineffectively, the same tools and technology can backfire. Instead of ignoring the challenges of big data, we need to predict, understand and overcome them.