Two years ago, Wired editor in chief and Long Tail author Chris Anderson wrote a provocative article entitled The End of Theory: The Data Deluge Makes the Scientific Method Obsolete. The article's message was that in the era of petabyte-scale data, the traditional scientific method of hypothesize, model, test is becoming obsolete, the victim of the combination of huge data volumes and the computer capacity to process them. “There is now a better way. Petabytes allow us to say: "Correlation is enough." We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.” Not simply content to take on the scientific establishment, Anderson seemed to go after  mainstream statistics as well: “At the petabyte scale, information is not a matter of simple three- and four-dimensional taxonomy and order but of dimensionally agnostic statistics.”

The university types I know – mostly statisticians, social scientists and business professors – reacted viscerally to Anderson's words. “Credulous” was the mildest rebuke in a line that included “bogus” and “bull----”. Three sample rejoinders:  1) Algorithms are useless on their own. In any data analysis, whether it is formal analysis or exploratory investigation, a key issue is how widely the results apply. Algorithms give no indication how or if results might generalize. 2) The neural net algorithms from psychology that were first promoted as a modeling panacea are now acknowledged as more limited. 3) Look even at the small yield of the massive data mining taking place in the national intelligence community (e.g., by massive screening of unselected phone call and e-mail data with dubious support from the Constitution). My bet is that “old style” human intelligence is far from obsolete in national intelligence.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access