Many business intelligence projects consider themselves successful if they simply get deployed into production. As the industry matures, more structure, consistency and a repeatable process for control and deployment of such projects is desired. It is necessary to establish a model for change and configuration management that is flexible enough to support frequent deployments and rigid enough to be documented and repeatable. The approach we recommend is tool agnostic and scales across BI and configuration management tools within the industry. This article highlights the configuration management processes successfully implemented for BI projects and shares some lessons learned in change and configuration management for such projects.
BI Projects Differ from Custom Transactional Software Development, but Configuration Management Goals Remain the Same
BI project implementations differ from transactional custom software development in various ways, including:
- No centralized code package to build and migrate because most of the projects are completed through the use of tools,
- Projects are data-centric, requiring significant data modeling, mapping and analysis and usually include more cross-project impacts in dealing with source systems,
- End users often access data in an ad hoc or self-service manner instead of through a static feature-driven interface.
While it is necessary to employ many of the same configuration management principles relevant to custom software development to BI work, the process for implementing these principles differs. In the BI world, tool-based configuration items include:
- Data models in modeling tools,
- Objects (mappings, workflows and jobs) in extract, transform and load tools,
- Query and analysis report structures or schemes in BI reporting tools and
These tools usually do not automatically integrate with an organizations existing change and configuration management tools, but this does not preclude their use for BI projects. Control, auditability, migration procedures for deployments, impact analysis and employment of best practices are vital configuration management concepts that still apply to BI projects.
Plan Early for Configuration Management Adoption to Ensure Consistency, Transparency, Stability and Control
Practitioners who understand the value of configuration management recognize that the organization can better trust what will be deployed to production when the organization controls the overall change process, documents and archives the necessary configuration items, and ensures a repeatable and consistent process from development completion through production deployment.
Once your organization has bought into the value that change and configuration management can bring, it is useful to have a starting point for implementation. Configuration management adoption should start with project planning that includes business, stakeholder and discipline constraints, time frames and key milestones. BI teams should initiate configuration management practices early in the lifecycle to enable the setup of configuration management as early as possible, so that governance and control can be phased in and have time for continuous improvement. The configuration management aspect of a BI project can be broken into three phases.
Phase 1: Project Setup
Setup starts with configuration and training of the change and configuration management suite of tools. If your organization has not determined the configuration management suite of tools, leverage configuration management industry research experts and articles to jump-start your selection process.
To ensure full traceability throughout the development lifecycle, we recommend you set up a requirements, testing, workflow/change management and source code repository for any BI initiative. Once the tools are in place, make sure to train teams on their expected use.
Phase 2: Configure Build Process
The term build within the BI world means employing the required steps for versioning of the artifacts intended for release and packaging them together.
Phase 3: Establish Enhanced Traceability
Configuration management processes establish traceability between artifacts from requirements to test cases to code changes. Most configuration management tools allow for traceability throughout the project lifecycle.
Configuration Management Best Practices for BI Require Process Understanding, Prototyping and Iteration
What follows is a list of critical concepts that will help you get started post-project setup and ensure your best chance for overall success:
Start small and adopt products as you refine the process. The planning process mentioned above is iterative. Start with one piece of your BI initiative to flesh out what works. This might be starting with one data mart, your staging area, an operation data store or even starting with one source system. Expect that the process you put in place will change, and make sure the documentation and communication channels are flexible enough to support iterative change.
Implement a consistent process for managing change. Change for BI initiatives often comes from a number of places, including business owners/stakeholder requests as well as impacts from source system changes. The group should decide how changes will be logged, managed, prioritized and updated and then ensure all changes are treated in the same manner. If your organization has a change-tracking tool already in house, it is a good idea to make use of this. If not, find a good spreadsheet template, freeware or shareware to use in the meantime.
Track impacts and dependencies. As you track change, make sure you have a way to also track and monitor cross-project impacts and/or dependencies to ensure coordination is proactive instead of reactive.
Have at least one resource dedicated to configuration management. You will clearly send the message that configuration management is a best practice and priority when you assign the role and set expectations. Depending upon the size and complexity of what you are trying to do, you may not require a full-time resource. The point is to ensure that at least a portion of someones time is dedicated to ensuring consistency and traceability, and that the agreed-upon processes are being followed.
Interview stakeholders to determine requirements and tradeoff priorities. Most projects require tradeoffs because most projects run into staffing or budget issues at some point in time. Knowing this, you want to interview management, architects, database administrators, analysts, modelers and developers early in the process to understand goals and priorities to better prepare your configuration management processes for handling tradeoffs.
Understand and control environment usage. Make sure you have support in separate environments for testing the deployment process in addition to testing functionality before rollout to production. Use established environments and employ release management practices that take deployment needs of BI efforts as well as source systems into account. Best practice says at a minimum there should be environments for development, testing (QA) and production.
Understand capabilities of tools. Knowing the capabilities of the configuration management and BI tools can help you maximize features and build a flexible configuration management architecture. This includes understanding:
- Export/import formats supported by BI tools. The creation of exported XML files can be a key for standardized implementation, because these files can be baselined in a configuration management tool and used for promotion via import to each subsequent environment.
- Vendor-recommended migration procedures for BI tools, recognizing there are also times to deviate from this.
- Version-control capabilities within BI tools. Most BI development tools have lightweight to no configuration management functionality built in to manage versions. When it exists, we recommend using the built-in versioning capabilities of the ETL tool to label together objects belonging to a particular release. This also allows for ease of follow-on automation.
- Command line/APIs. Understanding command-line and/or API functionality of the tools also affords the team an understanding of future automation capabilities.
Figure 1 depicts an ideal interaction between an ETL and configuration management tool. This method allows the project teams to develop code within the ETL tool and for the configuration management process to version code, trace change, and keep all artifacts of the release baselined together. It enables one version of the truth.
(To view a larger version of this figure see PDF below.)
Plan for automation. Planning for automation includes:
- Manual testing of the prototype process.
- Automating movement of code into the configuration management tool when possible. We recommend using standardized labels combined with the ETL tools query capabilities to coordinate being able to automate pulling code from the ETL tool and storing it with other baselines in the configuration management repository.
- Automating deployments to ensure release consistency. This is recommended only after enough iterations of manual deployments have occurred in order to shake out potential problems first.
Maintain baselines for backups and rollbacks. Because not everything always goes as planned, we advise having the entire snapshot of a release backed up and archived together. Doing so provides you an easy way of restoring to a prior version should a deployment not go as planned. If a configuration management versioning tool is in use in your organization, make sure all baselines exist here and that all artifacts that belong together are tagged or labeled together.
Ensure master source location for all artifacts. Because BI changes will be installed in multiple locations, determine the master source location for all artifacts early on. Communicate this throughout the team and make sure that code is archived and promoted to higher environments from this location. We recommend allowing the development master to reside in the BI tool while keeping the snapshot baselines for deployments and releases within the configuration management versioning tool. This allows teams to both develop BI changes more efficiently and easily stand up new environments when necessary from the well-organized stored baselines.
Prototype and diagram the prototype process; allow and plan for continuous improvement. The prototype process should:
- Follow a modified/agile development lifecycle with built-in quality checkpoints,
- Make use of a change control board and a qualified and consistent means of tracking and prioritizing change,
- Include deployment to multiple testing environments,
- Keep track of deployment instructions as change is promoted, testing migration procedures for the entire release as if it were the production deployment, to ensure accuracy of deployment instructions;
- Publish release notes for QA and end users and
- Be diagramed so it is easily shared and communicated with management and team members. (A simplified example prototype process diagram is shown in Figure 2.)
(To view a larger version of this figure see PDF below.)
Change and configuration management Can Enable Both Flexibility and Dependability in BI Implementations
Starting small and building your configuration management process iteratively allows your team the flexibility to make the quick progress your business users expect while still maintaining control over what is being implemented. Should your team desire to expand the span of change and configuration management control, you can easily take this to the next level in various ways. One option is to build in a diff process between the previous and the current release baselines and include in the process team verification/validation that the changes intended for the release made it into the baseline and that only the approved changes are in the new baseline. No matter how deep your team chooses to take your configuration management process, implementing all or some of the practices highlighted in this article will get your BI projects on track to more stable and repeatable implementations.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access