It’s wonderful that the BI toolkit is constantly expanding with new and better tools. Increased computing power and enhanced software solutions enable companies to deal with unprecedented volumes of data and provide access to information to practically anyone who needs or wants it. However, the wide variety of solution components to choose from makes the decision process to select the right tool all the more complex.
Additionally, with more users needing access, accommodating different types of users introduces further challenges when designing the solution. In short, more choices and more users often result in a solution design that provides less – as in less-than-satisfactory – alignment with business needs and goals. But by systematically assessing the key drivers of solution design, you can determine the solution that will truly align with your business needs, in spite of more choices and more users.
Let’s start by taking a look at the key solution components. Keep in mind that BI solutions are built to integrate data from multiple sources and make that integrated data set available for analysis. Based on this, our BI solution has three main components: data extraction and movement (usually referred to as ETL), data storage and integration (i.e., database) and reporting and analysis. Technology will need to be selected to handle each of these areas and to integrate with each of the other components as well. To select the most appropriate technologies for your environment, you will need to look at each of the key drivers and determine which technologies best address those drivers.
There are several other components that may be needed for your solution: metadata management, security, data quality and scheduling, to name a few. While this article focuses on the three main components, a similar approach should be applied for any additional areas of your solution.
So what are the drivers that will help us align our solution to our business needs? The categories I use are:
- Developer skills
- End user skills
Another way to think about these drivers is by considering the following questions: What does the solution need to do? What raw material (data) do I have to work with? What experience does the development team have in this area? How will people use the solution?
By combining an assessment of each of these areas with knowledge of the available technology options, the best choice can be made for each component of the solution. So let’s look at some of the most common facets of each of these components and consider how they play into your technology selection. By doing this, you will be well on your way to aligning your solution with the business needs.
The most important thing to know about your proposed solution is what it is intended to do. Does it need to integrate hundreds of sources of data? Will it be receiving real-time feeds that need to be analyzed, formatted and sent in near real time to end users? Does it need to handle petabytes of unstructured data? Will the solution be running numerous reports each week that will be emailed to a set of end users? How many end users will it support -- hundreds or just a few?
These questions will help you narrow down your choices to a manageable number. You should start by answering some of the bigger architectural questions, such as determining whether you need a traditional relational database solution or a data warehouse appliance solution. Do you want to have your hardware in house or in the cloud? Will your solution require a small data mart or big data architecture? Huge amounts of data, such as sensor data, could lead you to a big data solution. Producing a handful of accounting reports is typically a small data mart solution.
A BI architect will know how to read your requirements and help steer the solution in the right direction. Some of these choices will also feed into your technology selections as well, so be sure to include them in your selection process.
The list of potential questions for BI solutions is nearly endless, so it is important to be able to gather a sufficient set of requirements and understand how to categorize and prioritize them in order to design an achievable solution, rather than trying to nail down every single possible requirement under the sun.
You will need to identify which requirements will impact which of the main components of your solution: ETL, database and reporting/analytics. If you can build that list, you should then be able to match your requirements to the capabilities of tools in each of those categories. Once a short list of the tools is made, a more rigorous selection process can be launched that considers factors like price and user friendliness.
Data is the raw material that you will be using in your solution. The data will be brought into your solution (likely via ETL processes), conformed, integrated, aggregated, summarized, organized (in the data warehouse and marts) and made available for reporting and analysis (using reporting/analytics/data visualization tools).
The data must be assessed and understood in order to design an appropriate solution. The data will determine if your solution will fall into the big data category. Will there be enough volume, variety and velocity to qualify? Will the data be coming in real time or in periodic batches? Will data integration be complex or straightforward? Will the users understand the data or will it be too complex for them? The answers to these questions will help you determine the best tools to manage your data.
At a higher level, an overall data strategy also needs to be understood and addressed. The premise for the data strategy is that the source systems contain or create the data needed for analysis, and can move that data effectively to the data warehouse. A data strategy is especially important when implementing a major new software solution, especially an ERP system.
As with most solutions, start with the end in mind. Do your best to anticipate what analysis will need to be done after the new solution is implemented. Figure out what data will be needed to do that analysis. Then be sure the new solution can capture or create that data at the right level of granularity. After the new software has been deployed, you really don’t want to hear, “If you had told us you needed data at the daily level when we were installing the software we could’ve given it to you, but the system only captures data at the weekly level because of the way we configured it.”
One thing that often gets overlooked when selecting tools is the skill set of the existing development team. If you already have tools in house that your developers are familiar with, start by assuming they will be your tools of choice unless you can find a good reason to use something else. If a new tool is needed, people will need to be trained and, more importantly, gain experience with the new technology. You should also consider the degree of difficulty in training employees or finding new employees who possess the desired skill set. You could paint yourself into a corner by selecting a really nifty, inexpensive tool and then not being able to find anyone with the expertise to implement or support it.
End Users Skills
Remember that the best measure of success of a BI solution is the extent to which it is adopted by the end users. With that in mind, what is the profile of your end users? Typically, end users fall into three categories: those with little technical skill who only want to see completed reports; those who are capable of some manipulation and prefer parameter-driven reports; and those who are technically savvy, are intimate with the data and prefer to build their own reports and analysis from scratch. Find out what your user community looks like and select tools that they can use. Most of the today’s market-leading tools can accommodate all three types of users and it is simply up to you to build it out in such a way that it will accommodate your user community.
Because BI solutions are typically built over a long period of time with the primary goal of supporting business users, it is vital that the solution owner works closely and continuously with the business community to deliver a solution that aligns with both the business needs and the business capabilities. You want the solution to be used, so design a solution that makes it as easy as possible for your team to build it and the business community to use it. That way, they’ll be sure to come back for more.