The first thing I want to know when I see a new BI study is the identity and motivation of the main constituents – who's getting paid by whom. Often, BI thought leaders are hired to conduct “independent” surveys and summarize the results for BI tool vendors or the media. In a first scenario, the researchers are paid directly by the vendors; in a second, participating vendors may “sponsor” the analysis for the media who then pay the researcher. And of course there are the research companies themselves who are compensated directly by vendors to conduct studies in exchange for the buzz of the “findings”.
I have no problem with these business practices per se. As a partner in an open source BI consultancy, I'm all about profitable business and aggressive promotion of products and capabilities. Where I get heartburn is with the absence of reporting transparency that might mislead consumers on the validity of the research. It's one thing to shamelessly promote. It's quite another to cloak marketing under the guise of objective research.
As an illustration, one report I read from a research survey sponsored by a BI vendor had the vendor's data mining platform a close second to the R Project to Statistical Computing in utilization. I was a bit suspect since R has a world-wide cult following estimated at 2 million. Come to find out the the sample of respondents was primarily drawn from the vendor's mailing list. Go figure. A second example is a survey of visualization products by a BI media last year that didn't include a writeup of what many consider the top platform. It seems that vendor opted out – i.e. wouldn't pay a sponsorship fee – of the competition.
I'd like to see a simple set of reporting guidelines established for BI research. First, the economics of the entire proposition should be clearly articulated. If an “independent” thought leader is paid by a vendor to conduct a survey and publish findings in a white paper, that should be plainly visible to consumers. Similarly, if a research company or BI media subsidizes its work by soliciting marketing sponsorships from BI vendors, that information should be front and center in the study report. For sponsored vendor comparison studies, it's critical to know which vendors were invited to “participate”, and which “agreed” to be in the study.
Second, the research data acquisition process should be explained to consumers. For email surveys, knowing the “population” that sources the survey – what lists were used, where they originated, and the logistics of the email campaigns – is critical. In addition, it's important to understand the characteristics of the aggregate list in terms of basic demographics like company size, industry, job category, and BI maturity.
Third, characteristics of survey respondents should be similarly tabulated, contrasted with the population, and presented in the report. Differences between the sample and the population would shed light on potential response biases that could weaken findings.
Finally, either the survey instrument itself or a detailed description of the questions and measurement scales should be made available for review. Let the readers determine if the flow of the instrument or the questions themselves might be sources of bias that would weaken their conviction to study findings.
My thinking on how to improve the quality of BI research was given a boost when I came across Wisdom of Crowds Business Intelligence Market Study, by Dresner Advisory Services, LLC. BI veterans are certainly familiar with study author Howard Dresner. Now President of the aforementioned LLC, Howard was a trailblazer with BI in his capacity as Research Fellow at Gartner for years. He's also the author of two respected books on performance management.
WOCBIMS seems to address my big concerns about BI research head on. The Market Study was Dresner's creation with no up-front sponsorship or financial support (sponsors were invited after the report was completed). The study's modus operandi is clearly articulated. Readers can readily determine the what, who, when and how of this research in the early pages of the report. Characteristics/demographics of respondents are painstakingly detailed.
Instead of email campaigns, the study revolves on a “crowd sourcing” approach wherein a survey is made available online, then promoted using both traditional BI and social media. This method seems a credible alternative to vendor mailing lists for eliciting unbiased responses. “To publicize the study and garner support, we aggressively leveraged social media sites (e.g., Twitter, LinkedIn, Facebook), the press and existing email. Strong support was received from all corners, including key press: Information Management, Intelligent Enterprise, SearchDataManagement, IDG, Datamation, Smart Data Collective and numerous blogs and re-tweets. Vendors were also made aware of the study.”
Even with a number of cogent observations on an evolving BI industry, the transparency/disclosure of WOCBIMS's methodology is the report's most important contribution. That the survey methodology, design and population of interest, along with detailed demographics of respondents, are clearly articulated, is a major step forward for BI research. Readers have all the information they need to evaluate survey findings. That there were no vendor sponsorships of the work until after its findings were disclosed is also laudatory.
As positive as I am about WOCBIMS, there are areas for improvement in subsequent surveys. The study N of just under 500, which shrinks to around 300 for the 15 detailed vendor comparisons, is small. I suspect lessons learned on how to promote participation will yield better response in the future. In addition, it'd be helpful to see some of the psychometric properties of the survey instrument, especially the evaluation scales. Are indexes of “sales experience” and “consulting” as important as “recommended” and “value”? My guess is no. As a stats guy, I'm generally suspicious of forced rankings of computed survey scales. Means and medians are for research; rankings are for marketing. And I've yet to recover from the headache caused by the painful pie and horizontal bar charts that dominate the final report – enough grist for two newsletters from our favorite visualization scold.
As an unabashed supporter of open source BI, the Emerging Vendors category, which includes OpenBI partners Jaspersoft and Pentaho, is of special interest. But when I look at the ratings for these vendors, I have more questions than answers. What's the profile of respondents who note Jaspersoft and Pentaho? Do they primarily use the freely-available community edition or are they subscription-paying enterprise customers – and does that profile differ between vendors? As OpenBI has discovered over and over, there's a big difference between OSBI community and enterprise edition users. The former are often “guy in a garage” operations with no investment capital; the latter successful OEM's, well-heeled analytics-focused startups, and departments of mid-size organizations. The 101 respondents across 7 vendors in this category means an average of less than 15 per vendor, a small number for evaluative comparisons. Ideal would be 30 or more respondents per grouping.
Overall though, I give WOCBIMS a high A- grade and hope it's quality raises the bar for vendors, research companies and BI media to make their studies more methodologically sound and their approach more transparent. And I hope that readers, better informed by WOCBIMS, will demand this level of methodological quality in BI research going forward.
Steve also blogs at Miller.OpenBI.com.