Many people hear the word “Lean” and it conjures up images of featureless tools, limited budgets, reduced development and the elimination of jobs. Dispelling those myths out of the gate is crucial in order to garner support for implementing Lean BI from the organization and the BI team. If team members feel that by becoming lean they are working themselves out of a job then they will not support your efforts. If your customers feel that they will receive less service or be relegated to using suboptimal tools then they may not support your efforts as well.
So, what is Lean BI? Lean BI is about focusing on customer value and generating additional value by accomplishing more with existing resources by eliminating waste. Lean BI is a set of principles and practices that have been influenced by three main concepts:
- Lean manufacturing.
- Systems theory.
- Agile project management.
Implementing Lean BI is a journey, not a destination. The first step in implementing Lean BI is to understand that becoming Lean occurs incrementally over time, not all at once. It requires a change in not only the way in which most implement BI today, but also requires challenging some of the assumptions of the past. It can be as simple as following five principles.
1. Focus on Customer Value
Value is defined as meeting or exceeding the customer needs at a specific cost at a specific time and, as mentioned in my last article, can only be defined by the customer. Anything that consumes resources that does not deliver customer value is considered waste. Examples of waste in BI organizations include:
- Dashboards and reports that are developed but never utilized.
- Data integration processes that run but aren’t required.
- Indexes that are created on nonutilized database fields.
- Data that is retained on primary storage but never used.
- Rework due to outdated or incorrect business rules.
It can be difficult to define customer value since we tend to focus on the solution and the customer tends to focus on the problem. Sometimes we focus on providing solutions looking for a problem. For example, often BI developers attempt to anticipate what a customer requires based on past requests. It can often be the case that the BI team believes that it’s more familiar with the data of a functional area than the users are. Combined with the knowledge of the capabilities of BI tools, it is difficult to resist defining value for the customer. While value can only be defined by the customer, the BI team can help augment that value. For example, BI teams can demonstrate the capabilities of the toolsets, educate the users on when to use different types of graphs and charts and point out inconsistencies in the data. How do we know when something we do does not add value? Generally, it is under-utilized.
How do BI teams identify greater value up front and eliminate, or at least reduce the amount time working on tasks that, while emanate directly from the customer, add little value? Part of the solution is to instill a BI program governance model, which Steve Bell in the book “Lean IT” defines as the collective set of procedures, policies, roles and responsibilities, and organizational structures required to support an effective decision making process. The governance model is designed to link IT to the business by not only involving business users in the stewardship of the data, but also by having the prioritization of the work established by the executive team. It’s both a bottom-up and top-down process that engages both the business and IT.
The other part of the solution is to break down requests into projects and maintenance requests, and only work on projects that add customer value. For example, projects can be defined as any task or group of related tasks requiring more than eight hours of development time. For these efforts, ensure that they tie directly to customer value by identifying the result to be measured. Many BI practitioners will say that most projects can’t be measured, but in the book, “How to Measure Anything,” author Douglas Hubbard states “when a [person] believes something to be immeasurable, attempts to measure it will not even be considered. As a result, decisions are less informed than they could be. The chance of error increases. Resources are misallocated, good ideas are rejected, and bad ideas are accepted. Money is wasted.”
Another tool to identify value and reduce waste is to create values stream maps. Value stream maps help us to identify the value up front as well as eliminate waste along the process. The BI value stream is the set of all specific actions required to bring a specific project, process or task to completion. We can map each action along the value stream to help identify waste. (Click here for Figure 2, an example of a BI project value stream map.)
The first step in analyzing the value chain is determining the steps that do not add customer value. As identified by James Womac and Daniel Jones in “Lean Thinking,” generally, three types of actions are identified along the value stream:
- Many steps will unambiguously create value (VA).
- Many steps will create no value but are unavoidable (NNVA).
- Additional steps will be found to create no value and are immediately avoidable (NVA).
In this example, which is fairly typical of a traditional BI project, the most obvious area of waste is the time spent waiting. Approximately 31 percent of the total project time was time spent waiting for both signoff and resource availability. BI teams also should question the time spent during the detailed design phase. How much of that time is for creating documentation for the developers versus documentation for the customer and/or BI team? They should really question all the steps along the value chain to determine whether they add value and where they can be improved. In the short term, improvement may be achieved via rapid iterations, integrated testing, prototyping, and other agile practices. In the longer term, the implementation of development standards, common processes and procedures, feedback loops and business driven metadata repositories will likely lead to a reduction in waste and greater value.
2. See the Whole Picture
Learn to see beyond each individual architectural decision, organizational issue or technical problem by considering how they relate in a wider context. When business users make decisions and solve problems, they often only consider the immediate symptom rather than the root cause issue. For example, when a BI analysis uncovers rising call wait times in customer service, the answer is often to increase the number of agents rather than investigating the underlying cause for this increase. Consider how others may be affected by the decision, whether it be within our own department or cross-functionally. The cause of the increased wait times may be due to poor product quality from the introduction of a new supplier.
For example, consider this discovery at a recent consulting engagement. Based on the result of a BI analysis, the purchasing department decided that in order to save time entering purchase orders they would create blanket orders that started with a quantity of 10,000 and receive a portion of the total order for each delivery. Unfortunately, the system did not allow the purchasing agents to schedule parts of the order or the ability to update the price after the order was opened. The result was that while it saved time for the purchasing group, it created much more time to resolve receiving issues and led to a tenfold increase in pricing discrepancies. In addition, they were unable to rate the performance of their suppliers, ultimately causing on-time delivery issues to the customer. In this case, BI was used to optimize a local process at the expense of the broader organization.
This is also a challenge on the technical side of BI. For example, the same data integration jobs fail consistently and the response is usually to simply restart the job rather than determining why the job failed. It could be that the backup job of an upstream system is running into the load window, in which case it’s likely that the number of failures will increase over time. In addition, architectural decisions that focus on solving immediate issues, such as poor query performance or long data load times, often lead to larger problems over time. For example, changing the network backplane between database nodes might solve throughput between servers but end up creating greater processor and memory bottlenecks. When we fail to consider issues in a wider context we tend to optimize only the local problem at hand at the expense of the larger system and often don’t address the root cause of the issue.
3. Iterate Quickly
It is often the case that by the time a project is implemented, the requirements have changed and part of what is implemented is not required anymore or is no longer a priority. When features, reports and data elements are implemented that aren’t utilized, it is considered waste. There are two main levers to iterating quickly: increasing flow and reducing the scope of each iteration.
There are many steps to creating flow, such as incorporating the business into the development and design process, implementing test-driven development, creating level demand and developing multifaceted BI resources. The key is to not only reduce the amount scope and minimize manual processes per iteration, but also reduce the number of handoffs throughout the process. The more handoffs and interruptions that are created, the more time that is spent by resources simply trying to get up to speed on what’s been done in the previous step or steps. It also tends to be more difficult to control quality the more people that are involved in the process. In the most extreme example, if you have one person that performs all tasks in a project then you know which resource created the quality issue and you can help that resource improve the area where the issue was created either through additional training or a change in the process. While it’s improbable to train all of your resources to fill every role, having resources that can fill more than one role helps to reduce handoff, reduce errors and even can lead to a more satisfied staff.
When the scope is reduced, it doesn’t necessarily mean that you need to reduce the scope of the entire project, just each iteration. BI teams can work with users to help identify the minimum number of features and capabilities that will meet the initial requirements. This is referred to by the Lean community as the minimum marketable feature set (MMF). Once the first iteration is delivered, the business users will be able to make more precise requests about what additional functions are needed in the next release cycle. This is continued, ultimately culminating in the broader solution. Communicating this to users helps to ease their concerns that their project is being curtailed.
4. Reduce Variation
Variation in BI is caused by a lack of standardization in processes, design, procedures, development and practices. Variation is introduced when work is initiated and implemented both inside and outside of the BI group. It causes waste in a number of ways including the added time to reverse engineer what others have developed, recovering ETL jobs caused by maintenance overlap, the extra time searching for scripts and reports, and the duplication of development caused by two developers working on the same file. Examples of variation that cause waste include:
- Different definitions for the same element.
- Data formatting differences.
- Different business rules across applications.
- Reports named differently in the BI portal.
- Scripts saved in different locations.
Often, variation is introduced by external consultants, especially those working offshore. Most consulting companies have their own methodologies and standards, and most consultants have their own styles of development. This can lead to different naming conventions, mapping strategies, error handling and countless other areas that produce confusion and added maintenance burden. Make sure to communicate your standards, processes and practices at the beginning of an engagement and ensure compliance during design reviews.
Standardization doesn’t mean “unchangeable” nor does it mean “stifle creativity.” Consider adjusting your standards when new ideas are introduced that make sense in your environment. Ensure that your team understands that the standards aren’t written in stone.
5. Pursue Perfection
Perfection is a critical component of Lean BI even though the key to successfully pursuing it is the understanding that you will never get there. The key to pursuing perfection is to focus on continuous improvement in an increment fashion. The difficult part is that BI teams tend to be inundated with work and don’t have time to spend working on small internal projects. The key is that the first Lean initiative should free up enough time to work on the next Lean initiative.
There are situations where compliance requirements make changes to the environment very difficult and time-consuming. For example, pharmaceutical companies are regulated by the FDA and many reporting systems are required to be validated if utilized for product safety decisions. This can add significant overhead for even simple changes. In this situation, consider choosing BI tools that have better change auditing capabilities and change traceability.
The goal of BI is to provide customer value by facilitating the process of turning data into information, communicating that information effectively and leveraging it to enable better decisions. These decisions exist at all levels of within an organization, and can be both inward facing and outward facing to IT. These decisions can also range from strategic to tactical. At the senior management level information is highly summarized, often into scorecard and trend views, so broad patterns of performance and behavior can be monitored against strategic goals. At the middle management level information is more granular but still summarized, helping to identify emerging problems and weaknesses to be addressed.
At the individual and team level, information is often highly granular and transactional, helping to keep an eye out for problem prevention rather than reaction. As you move down these levels, the data latency requirements decrease, the data volumes increase and the analytic capabilities broaden. Delivering these levels of information requires a strong business alignment and the agility to keep up with changing requirements. Lean BI principles help organizations deliver greater value across all levels of decision-makers by reducing waste, focusing on the customer and never settling for less than perfection.
Steve Dine is the managing partner and founder of Datasource Consulting, LLC. He has extensive experience delivering and managing successful, highly scalable and maintainable data integration and business intelligence solutions. Steve combines hands-on technical experience across the entire BI project lifecycle with strong business acumen. He is the former Director of Global Data Warehousing for a major durable medical equipment manufacturing company and currently works as a consultant for Fortune 500 companies. Steve is a faculty member at The Data Warehouse Institute and a judge for the Annual TDWI Best Practices Awards. He teaches courses and presents on the topics of Lean BI, BI in the Cloud and Enabling BI for the 21st Century. Steve earned his bachelor's degree from the University of Vermont and a MBA from the University of Colorado at Boulder. Contact Steve via email at firstname.lastname@example.org or on Twitter: @steve_dine