Born in 1918, my elderly neighbor grew up during the Depression era and loved the independence that working and driving offered throughout her life, although she never did learn to fill up her gas tank. One didn’t want to take independence too far, after all. Besides, she didn’t have to — her late husband, a WWII army veteran, always insisted that it was his duty to do it for her. When he passed away, her co-workers at the plant where she worked took over this chore, followed by her boss who continued the tradition after she retired 30 years later.
This story illustrates a larger point about embracing change — while willful ignorance can have its benefits in genteel society, executive leadership that maintains willful ignorance of their own data management and business intelligence operations can cause serious damage to their enterprise’s bottom line. With the advent of breakthrough hybrid technologies that employ both standalone data warehouse appliances and Hadoop, now companies can better fuel their own BI engine in today’s challenging economic climate.
Begin by galvanizing the troops. In terms of improving big data, data warehouse and BI solutions, many companies still operate under an archaic, “let the IT guys/software provider/business unit deal with it” mentality. However, I’ve found that the most successful businesses are able to fill up their own big data gas tank and leverage it efficiently. How? Emerging advances in leaner, hybrid BI methods and architectures are currently gaining a lot of traction, successfully moving clients toward powerful data management and BI solutions. This technology is similar to the hybrid vehicle concept that endeavors to wean our population off of our dependence on fossil fuels. Hybrid cars are a pragmatic, cost-effective approach automakers have adopted in response to 1) increased consumer demand for more practical, fuel-efficient vehicles; and 2) newer, stricter government regulations aimed at countering documented increases in greenhouse gas emissions.
An enterprise’s “Three Vs” of data — volume, velocity and variety — are what determines the types of technology that are warranted. If my neighbor were still driving today, I wonder if she would ever consider an electric car with a charger in the garage. Why? Because her “three V” threshold has always been pretty low; all of her driving was short, local trips within 10 miles of home. Similar to my neighbor, organizations need to evaluate their three Vs and accurately determine their threshold to embrace new approaches. Calling these approaches “lean” solutions, they take advantage of new tools, technologies and processes to give businesses much faster access and overall control of their own data, in real time, streamlining business processes and reducing the total cost of ownership.
Data warehouse appliance (DWA) technology is ideal for businesses that have three-V thresholds that are also under certain data parameters and guidelines. A “lean” DWA approach involves the acquisition of all available data, not just slices of data, and applies principles of real-time computing to allow permutations of that source data, using virtual source-of-record images of the data, not the physical data itself. The amount of required data transformation layers is reduced from over 10, in many cases, to three steps — Source Of Record, Insight and Consumer — greatly decreasing complexity and overall processing times. By employing these lean BI techniques, DWAs offer performance advantages and cost savings that can’t be denied.
Rechargeable electric cars offer performance advantages that include high efficiencylow emissions and less noise, although the sticker price may be a bit higher and travel distances have a limited range. Likewise for new BI consumers, a scalable data warehouse appliance can really go a long way in getting your business over those first three-V hurdles. Using lean concepts with DWA technology, rapid changes are straightforward, overall project costs are lower, due to simpler design and less need for data modelers and developers, and maintenance is much less complicated. Lean DWA approaches also offer increased functionality, since all data is available. Business units can consume information on their own terms, when they’re ready, without having to depend on other business units.
Can an enterprise’s three-V capacity outgrow a DWA solution? Yes and no. As a rapid-deployment solution, DWAs are also highly flexible, designed to adapt to an organization’s growth and increased need for the three V components of big data. However, there is a natural inflection point in the life of a DWA-based lean solution, when significantly more volumes and variety of data will be needed at faster speeds. The best new BI solutions embrace the concept of hybrid technology. Hybrid BI solutions combine new versions of traditional BI architectures with new technology like DWAs.
Gain the most benefits from a hybrid lean BI solution. What if my neighbor added more, longer destinations to her routine, for example, a visit to her sister who lived 25 miles away? When this distance proved too much for her, an add-on solution could be called for in the form of a taxi, limo or an accommodating friend. Problem solved. For those of us who travel longer distances as a rule rather than the exception, however, the better bet would be a hybrid car which combines a new, high-efficiency gas engine with a rechargeable, electric battery component. High-efficiency, gas-powered engines — which leverage older internal combustion technology in new, more efficient ways — are still better suited for heavy, long-distance driving and hauling, while electric power offers better performance and efficiency on shorter distances.
Right now, the best BI solutions successfully balance the benefits that proven traditional processes offer with newer technologies that foster self-reliance and don’t drain valuable resources, energy or time. Hybrid BI solutions employ “electric car” DWA technology to take advantage of its more nimble and single source approach for lower three V processes. To accommodate big data applications that have much heavier processing demands that a DWA couldn’t handle or become too costly to scale, hybrid BI solutions add one or more technologies (like Hadoop), which can tackle more complex processes on less expensive commodity hardware. Hadoop, like new highly fuel-efficient engines that that are based on yesterday’s internal combustion technology, is much more cost-effective than a DWA to pre-process semi- or unstructured data and store massive amounts of big data.
The ultimate hybrid lean solution would give an enterprise the ability to drive and maneuver its three V threshold through all kinds of terrain at top speeds while conserving energy at the same time. Can it be done? Absolutely, with the keys being cost, processing requirements, and the three V threshold. Let’s consider the cost factor first. As more and more data is brought into your enterprise data warehouse (whether it is a DWA or traditional database), the footprint of the database increases, which drives costs up and impacts performance. Next, let’s factor in the processing requirements. Not all processes need to be executed within your EDW. In fact, storing unused data in an EDW is just a bad, wasteful practice — sure, it’s easier, but much more costly. An enterprise should complete an audit of all its processes to determine if it makes more sense to bring certain data and processes into its EDW, or execute it within in a pre-storage and pre-process Hadoop environment.
Of the three V’s, variety typically means semi- or unstructured data, which a database neither stores nor processes very well. As the variety of data increases, the demand to take the unstructured data and structure it in a pre-process becomes greater. This pre-process is another good example of how to leverage a big data Hadoop environment. The other two V’s — volume and velocity — are all about scale and processing power. Again, the tipping point should focus on the following test: “Should I process and store this volume and velocity within the EDW environment, or better yet, pre-process and pre-filter this data, and only bring in the data that is required — already analyzed, compared and evaluated — into the DWA?”
A truly balanced hybrid lean framework optimizes total cost of ownership by maintaining speed, functionality and flexibility. Adding additional architecture allows for exponential increases to processing power while significantly increasing amounts and types of available source data. If your business finds it would gain more measurable and meaningful intelligence from integrating these larger stores of unstructured, semi-structured and structured data, hybrid technology provides much needed access to proven ETL tools and Hadoop technology, greatly extending the life of your EDW technology investment.
Widowed in her 40s, my neighbor always bought American-made cars to honor her late husband’s military service. He swore by Pontiacs and Buicks, and that was always good enough for her. While Pontiac is no more, Buick’s website describes its 2014 Lacrosse performance and fuel efficiency this way: “Who says luxury and performance can’t coexist? Opt for the 304hp V6 engine and you’ll be enjoying the road in no time. Innovations like eAssist technology help conserve fuel (EPA estimated 25 mpg city/36 mpg hwy.” Similar to how U.S. car manufacturers like Buick are making much better use of the earth’s energy resources with hybrid vehicle innovations, businesses that work to understand hybrid technology’s long-term advantages and deploy appropriate hybrid solutions will gain much better governance, internal control and overall use out of their big data assets. Hybrid technology affords enterprises a better opportunity to grow and prosper in what seems a perennial challenging economy.
Are better solutions in our future? Of course. Innovators from all walks of life, not just automakers and data scientists, are always driven to seek what’s beyond the next hill when it comes to developing better-use strategies for dwindling resources, while doing no harm. In the meantime, incorporating hybrid BI technology can satisfy your company’s need for better business intelligence — for the foreseeable future — with more user-friendly, agile frameworks while further advancements in BI solutions are developed and refined. Don’t forget, though, to make sure you have a full tank of gas and your battery is charged — and that you know how to do these tasks yourself if you have to.