The recent momentum of big data and related advanced analytics capabilities has captured the attention of executives and IT professionals across all industries. But in order to not be swept away with all the hype, it is helpful to take a step back from the recent trends to revisit five key principles around data warehouse design. These fundamentals are intended to help organizations that are in the early planning stages of a data warehouse or organizations that have an existing data warehouse but want to review their own design and direction.
1. Define your warehouse roadmap.
Business and IT alignment is critical. Both groups must define and agree on a program for the warehouse. Implementing a warehouse is not a one-and-done effort. Continual growth and refactoring will occur as business processes change, data volume grows and technology capabilities increase. Not defining a roadmap for evolving the warehouse upfront increases the risk of creating an environment in which the warehouse will become stagnant or unmanageable.
2. Know your data.
This sounds simple, but it is frequently ignored. Knowing your data goes beyond defining how your source data entities map to your target data entities. Knowing your data requires a detailed level of source system analysis. It is imperative that a deep-dive analysis of the source system is performed to validate, confirm and define all the necessary data rules. This level of analysis will also drive an understanding of the quality of data available. GIGO (garbage in, garbage out) is not a methodology for a successful data warehouse program.
3. Invest in the right amount of scalability.
This principle is related to the definition of a data warehouse roadmap. Scalability is often an issue when it comes to data warehouse design. All too often, proper modeling techniques are not used, causing the data in the warehouse to be replicated, repurposed or overwritten. To this point, it is valuable to understand the potential growth related to your organization and the industry in which you operate so that you can incorporate the right amount of flexibility into your model to account for future needs. Note: The right amount of scalability is the key. If you are in a slow-moving industry, do not model your warehouse as if you are in a retail industry.
4. Know your resources.
If your team is short on data architecture or data modeling expertise, it makes sense to augment your team with an additional resource proficient in those skills. If your team has a vast amount of experience in a specific platform, consider that as factor when determining what your strategic platform will be. Understanding your resources will allow you to plan development efforts more effectively.
Keep it simple, stupid. While understanding the need to design something that is scalable, simplicity in design is also a key component of a good data warehouse design. In some manner, this principle impacts all the other principles. Understanding how your data warehouse will mature and recognizing that there will be an evolution should reinforce the fact that the design should be simple. An overly complex design that is difficult to manage will only get worse as the warehouse evolves. Knowing the data provides you with the insight required to keep your model simple and clean. Knowing your resources should be a reminder not to over-engineer your warehouse unless your goal is to build something that your team is not equipped to support.
These five fundamental principles are key to defining the proper strategy for your data warehouse and are the pillars for establishing a solid foundation. Defining the initial direction and vision for the warehouse, along with an introspective review of your organization, business and resources, will go a long way toward the success of your data warehouse program.
Editor’s note: A follow-up article focusing on five principles related to business intelligence fundamentals will be published October 15.