- Data sourcing
- Data extraction
- Data integration
- Data cleansing
- Data aggregation
- Data modeling (star schemas, cubes)
- Metrics management
- Queries
- Reports
- Dashboards
- Alerts
- Delivery (portals, schedulers, emails, etc)
For years there were many attempts to automate some of these steps via metadata. So rather than than coding source to target SQL transformations or DDL for DW generation vendors came up with, what I know call "1st generation" metadata driven BI tools, such as:
- ETL tools where metadata auto-generated SQL scripts for data extraction, loading and transformation
- BI tools where metadata auto-generated SQL for queries
- Data modeling tools where metadata auto-generated logical data models and DDL for physical data models
But, the "2nd generation" metadata driven BI apps (note apps vs tools now) do much more. For example, they:
- Use metadata to generate multi vendor apps (like BalancedInsight, Kalido and BIReady do), and having a single place where changes can be made
- Use metadata to generate all three (ETL SQL, BI SQL, DW DDL, like Cognos, Wherescape, BIReady do), and having a single place where changes to all 3 can be made
- Using metadata to generate report layouts (like Cognos does)
The bottom line here is that these 2nd gen metadata driven BI apps actually generate apps (vs SQL).
I am currently looking at the following vendors ...
- ASG
- BalancedInsight
- BIReady
- IBM Cognos BI Applications
- Kalido
- Modulant
- Wherescape
... capabilities to see how many steps out of ...
- ETL source to staging
- ETL staging to DW
- DW DDL
- Cube generation
- Semantic layer generation
- Report generation
... they can auto-generate via metadata. Stay tuned for a full report.
Boris also blogs at http://blogs.forrester.com/business_process/.
However with this approach:
- Data is copied from various sources to another holding place: A new super database comprised of all the source databases (data warehouse). - Years of man hours are spent in mapping the various data sources to generate one consistent super database (data warehouse)
I recently came across a new approach to the data problem for BI. With LinkWex Maestro, data from disparate sources, including web services, are unified into a virtual data warehouse. Maestro normalizes disparate data sources, links them using our unification engine (no manual mapping) and then virtualizes the combine data to be serviced via our extensive set of Application Programming Interface (API). No copying, No replication, and No expensive Hardware.
Check out www.linkwex.com.
Intersystems(www.intersystems.com) has a line of products that address some of the issues you mentioned.
I've been following meta-data agility since 2004 and think the relational model is starting to really feel the stress under agile and fast changing business needs of today.
-Mahinda Gunasekera