Standing Up for Agile BI. (Sort of.)

The theme of last week’s TDWI conference in San Diego was “Creating an Agile BI Environment—Delivering Data at the Speed of Thought.” I sat in on some workshops expecting to learn something new about agile BI, if not eat a little crow. You see, I’d received mixed responses on my assertion recently that agile BI attracted companies “that don’t have the discipline to enforce BI development in the first place…” Turns out I did a lot more of the former than I did of the latter.

Without fail each TDWI instructor acknowledged that agile was borne out of operational systems development and needed to be revised to support BI. Each emphasized time-to-delivery. But the messages diverged when it came to adapting agile development for BI.

In my cynical moments I’m convinced that agile is for extroverts who’d rather spend time talking than developing or, God forbid, documenting. They like the stand-up approach to requirements gathering. The efficacy of agile BI in practice is very much still up for debate. TDWI’s instructors focused mostly on how to wring as much productivity as possible out of BI development tasks, rightly citing data modeling and ETL as the main time-zapping culprits. But I was struck by how little content focused on sustained business engagement.

I didn’t get to stay in Larissa Moss’ workshop long enough to hear her formal take on agile BI but I suspect she’s got the right idea, which is (my words here): Think big, scope small. For years we’ve admonished clients that the success of BI programs is all about how the projects that comprise them are sliced. We tell them to embrace the reality of multiple releases of their BI applications. This keeps it warm with the business. It ensures regular delivery. It lets BI teams portion out critical data an app at a time. Users grow accustomed to seeing new data and using new reports. You get the idea.

So no new surprises here. Except for the sophistication of people who’d tried agile BI. These people know their sprints from their sandboxes. I was especially gratified by the smart comments coming from my trusty blog readers. Here’s a sampling of some of the more trenchant comments on my previous blog post:

• One of the biggest challenges I have faced is getting people to actually think in 60-90 day chunks of work. Whilst it is easy for people to envisage their ultimate goal, defining smaller steps to get there seems to be a skill which is lacking, especially in those tasked with defining the scope and tasks for each "chunk". -Rhonda B.
• Agile should not be used as an excuse to work without discipline. It is a different kind of discipline and it requires planning and thought like any other exercise but it should not be an excuse for bad behavior or short-sightedness. -Michael S.
• … the degree of unpredictability of the [BI] effort is in inverse proportion to the extent, degree, and clarity of the documentation … -David S.
• Please don't bring usual operational system techniques and methodology into the BI world. …BI is all together a different ball game, and let it remain that way. -Rahul H.
• There is no EASY button for DW/BI. Effective data management requires hard work, rigorous engineering standards, and organizational discipline. -Bob Conway
• It will only be through adding more data, adding more functionality and even retiring outdated elements that ongoing value optimization can occur. -Kelly Lautt
• If the result of any BI development activity does not directly contribute to improving profitability, growing revenue, increasing market share and retaining customers, then it surely matters not a whit if the processes are "agile" or otherwise. -@PeterBeddows

Agile BI works when it’s sober, with managed expectations, realistic end-users, and experienced developers. But it breaks down when the goal of accelerated development time usurps delivering business value. From the messages last week it seemed that increased speed was the main motivator behind agile BI.

I haven’t changed my mind about agile. It seems to be embraced most ardently by IT people who have been unable to engage the right level of business executive in the strategic conversation. No one admitted it outright but I suspect some BI teams have spent more time honing their agile development approaches and researching BI “accelerator” tools than they’ve spent educating their business users about why BI is different in the first place.

But these users need to know that BI can be a strategic game-changer. That it can foster an early-mover advantage into new markets and increase competitive nimbleness. That it can streamline business processes, drive economies of scale, increase revenues, and delight customers. Sometimes these conversations require formal education, missionary work, debate, and even rigorous long-term planning. And that might just mean sitting down.