Information Management Discussions

[> ScottWAmbler]

I'm glad to see this sort of article published. The reality is that agile approaches have a lot to offer the data community, and the data community clearly has a lot to offer the agile community once they determine how to interact with agilsts a bit more effectively (yes, there's a significant cultural impedance mismatch still). At the agile data site, www.agiledata.org, is a wealth of material regarding agile strategies for data activities.

[> Chris Gerrard]

I've been practicing agile BI for over twenty years, and am very happy to see this article. Far too many BI initiatives fall into the trap of trying to deliver the universal analytical appliance as their first achievement. As the maxim puts it: "You can't start with everything; if you try you'll never deliver anything."

Agile BI, which includes agile data modeling, deliberately set outs to deliver business value early, often and continuously in the form of meaningful, timely, high quality analytics.

While I agree with the major thrust of the article there are a couple of misinterpretations of agility that it perpetuates.

First up: "A majority of projects driven by agility, however, lack a big-picture focus and often strive to deliver small slices of functionality within tight time frames, at times redoing and revamping prior work." There are a couple of corrections required to this statement.

Redoing and revamping prior work is an essential ingredient element in any agile undertaking. In "normal" agile software development this is called refactoring and is thought of as a good thing. Agile BI is no different. In fact, refactoring is always part of the process - agile projects surface and embrace it; big-BI project hide it under the covers where it pollutes everything and invisibly hinders progress.

There a hint in the statement that agility projects bias is towards a "lack a big-picture focus". (yes I know it's qualified but the smell of allegation lingers). This is a misrepresentation of the true nature of agility. True agility requires a professional awareness of the large scale structures of the diverse elements in the mix. A project that doesn't do this may wear the clothes of agility, but doesn't possess its soul. Just throwing functional bits out into the world without regard for the rich complexity of the environment will inevitably lead to a terrible mess. And an organization that creates such a mess is unlikely to get it cleaned up.

There's also an argument to be made against using a pre-existing framework to guide agile BI. While it's incumbent upon the BI professionals to recognize the conceptual architectures-business and technical-forming the environment, the paradigm that a pre-formed architecture is necessarily a valuable asset fails to recognize the costs, burdens, and downstream ill effects that result from the principle of simply trying to "fill in the details." Experience shows that the best architectures are emergent, resulting from the continual refactoring of the developing systems while adhering to suitable and appropriate architectural principles. In this approach, there's always an intentional architecture that fully supports the real needs of the system, is flexible and adaptable (agility at work!), and can be grown and refined as necessary while never imposing costs not directly related to delivering specific business value.

[> ScottWAmbler]

Just wanted to point out that the agilists lack an understanding of the big picture is a bit of a myth. One of the things that I do is run industry surveys to find out what's actually going on out there as opposed to believing the rhetoric that gets offered up, particularly on the web. Several surveys have shown that close to 90% of agile project teams invest some time in up-front requirements and architecture envisioning, activities where the "big picture" would very likely be discussed.


Also, refactoring is an investment in quality. When you find something that isn't of sufficient quality you invest the time to clean it up. It's easier to work with high quality code and data sources than low quality ones, so refactoring often speeds up your development efforts in the long run. And yes, contrary to popular belief, it is possible to refactor databases and it is quite easy to do so.

[> Chris Jackson]

Here's the dilemma for anyone concerned about data architecture, strategy or governance - and that should include anyone running a major cross-functional systems or change programme:
i) You know you need a shared, enterprise-wide data picture in order to understand the pinch points and risk areas, and just what it is you're all arguing about.
ii) Enterprise data models as embraced in the early 90s were white elephant projects which ran into the sand and became career-damaging.

The approach we use is to start from the top, with the business:
Create a subject area model which is deliberately very generic and not functionally aligned - subjects like party, product, location, resources, agreement, engagement. Stay between seven and a dozen areas.
Drill down to a conceptual model, which breaks the subjects out into the key data concepts meaningful to the business, and the relationships between them. No worrying about cardinality, normal forms etc at this stage. Probably no more than 120 concepts to cover a large, complex business.
Produce these rapidly from interviews across the business, understanding their stories. It takes us about a month to reach a pretty solid V1 for a business we're not familiar with. But avoid paralysis by reminding yourself and everyone else - this is V1. You are allowed to refine, tweak and extend as the models are put into use supporting your projects and data governance initiatives. A V2 is inevitable.

Now you're in a good place to move on to an enterprise business dictionary (built in stages), an enterprise logical data model (built in stages), master data management design or a broad systems and data architecture.