A June 2012 Chicago Tribune article by Phil Rosenthal entitled“This Man Knew How to Size Up Consumers” highlights the 40+ year career of David Wallerstein to illustrate this point. Mr. Wallerstein, a friend of Mr. Rosenthal’s grandfather, was notorious for telling outrageous stories involving his influence with Walt Disney and Ray Kroc, among other notable entrepreneurs. After his death 20 years ago, friends discovered that he didn’t make it all up. His obituary backed up his stories; for instance, as a long-time operator of movie theaters, he was the man who introduced butter on popcorn, as well as ice in soft drinks. As a board member of McDonalds, he also convinced his friend Ray Kroc to add large size fries to McDonalds’ menu. Mr. Wallerstein was seen as a brilliant pitchman who relied on assimilating his creative instincts with his observations of consumers’ tastes and behaviors.
Fast forward to today — our global business environment is too large to depend solely on intuition. Now business intelligence, which can tell us what consumers are buying and what influences them, is obtained at warp speed through the analysis of mass quantities of both structured and unstructured data — big data. What’s the new constraint? Creativity, which Mr. Wallerstein had in spades. In today’s data-rich society, business users grapple with how to fulfill the promise of big data through innovative approaches to analyze and leverage it effectively.
As far as big data is concerned, businesses could use more of Mr. Wallerstein’s creativity, particularly relative to big data. For years business users have been saying, “The things I could do if I got my hands on all of that data!” But throughout this period, too many technological and financial barriers stood in the way of big data’s potential. From acquiring too much data to store and process to the fact that the data was inherently unstructured, the algorithmic gymnastics required to integrate it were extremely difficult and cost-prohibitive. With all of the recent advances in technology, businesses can now remove these constraints and take advantage of big data in a new way.
What is Big Data?
In today’s global marketplace, deriving higher quality solutions to solve challenges in business, health care, finance, education, agriculture, retail and other industries now requires the power of big data analytics. The promise of big data analytics is that it can organize, quantify, and make sense of virtually all types of random data deviations to help decision-makers better understand an enterprise’s needs and goals.
Big data was so named because of the sheer volume of data currently being processed throughout the computing universe. While many disparate types of data have been around for years, until now this raw information hasn’t been harnessed or structured effectively. As our society evolves toward the use of more technology (PC, tablets, smartphones), more data is generated (e-shopping, e-transactions, e-reviews, social media, etc.) that could be used to great benefit. In the past, limited technology, combined with high costs, detracted from an enterprise’s ability to process data and design products to analyze it; it had to be structured within an inflexible database format and the costs of programming or developing a new analytic structure were so high that the expense was rarely justifiable or profitable.
Why is big data so prevalent now? The “Aha!” moment occurred as the market found a way to leverage commodity hardware and software technology, helping enterprises process this information quickly, cheaply and effectively. For example, now a terabyte of data can be stored on a flash drive. Processed data can be reduced from thousands of dollars per byte to hundreds, or even tens of dollars. Data mashups can be executed on the fly in reporting tools. All these advances allow enterprises of all sizes access to big data analytics so that they can make more focused, informed decisions.
Big data can be classified into one of three types: structured data, semi-structured data or unstructured data.
Creative Tools and Techniques to Gather, Process and Integrate Big Data
It’s important to remember that it took David Wallerstein many years of first-hand observation to understand his consumer target market before acting on that knowledge. Like Wallerstein, most human beings have a similar cognitive capacity to store, retrieve and process seemingly disparate bits of unstructured information and events, form conclusions from this information and then make educated decisions. The trick in today’s big data world is to use that same creativity to develop rule-based toolsets that can rapidly do what our brains do naturally: store, retrieve, process and integrate all of the structured, semi-structured and unstructured information that is currently available, to form an analysis and make educated decisions which can further an enterprise’s goals.
Each of the data classifications defined in Table 1 requires specific toolsets to process, organize and combine it into a fully integrated big data database and drive out valuable analytics.
Imagine the possibilities if someone like David Wallerstein had these dynamic big data analytic tools at their disposal. For example, one innovative toolset for semi-structured and unstructured data is comprised of a rules-based engine which converts voice to text from video and audio data, and puts it into semantic context for integration with more traditional, structured data. An enterprise can capture completely unstructured content, run it through bundled commodity PC hardware at lightning speed, and create a data-rich treasure trove that is fully compatible for integration with valuable business applications.
