When I first arrived to the work world 30 years ago, I encountered two roadblocks to my enthusiasm for mathematical and statistical optimization techniques in business. The first was a data management problem. Collecting, integrating, organizing and manipulating data was a very thorny, sometimes intractable task, consuming almost all analytical energy. Technology to facilitate data management, both hardware and software, was just beginning to evolve from the mainframe/COBOL/network database paradigm. Machine cycles were rationed, storage was scarce and expensive, programming was low level and data was unreliable. Probably 98 percent of statistical effort revolved on building trustworthy data sets. While data quality issues persist to the present, many of the other hardware and software problems have been solved. In fact, the ascent of mini/micro/personal computers with UNIX and Windows as well as the emergence of relational databases should probably be heralded as fundamental enablers of modern business intelligence.

A second impedance was more subtle. I came out of school very excited about the rigorous statistical techniques I'd learned. Linear models, multiple and logistic regression, categorical data analysis, time series models, econometric models, multivariate techniques such as discriminant analysis, canonical correlation, etc., certainly worked well in research settings; why was it so difficult to make the translation to the new demands of business? Was there something fundamentally different about analysis in a business setting? Was the rigor of research missing in business? My conclusion was that I was dealing at that point with a different problem domain. The statistical models I'd learned focused on testing or confirming hypotheses, whereas in the business world I needed approaches for developing or formulating hypotheses. Techniques to help discover relationships were missing. Thankfully, help was on the way.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access