MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) today announced a new initiative, bigdata@CSAIL, to explore the solutions and challenges surrounding the emergent field of big data.

The program makes partners of MIT, the state of Massachusetts and Intel Corporation, which will be establishing the new Intel Science and Technology Cennter facility for bigdata@CSAIL. It brings together leaders from academia, the technology industry and government to explore and develop techniques for capturing, processing, analyzing, storing and sharing big data with a goal of “making it more useful for society as a whole.”

The term “big data” commonly refers to data sets too large or complex to address with conventional tools, and is a byproduct of the technology and information booms of the last decade. The field is being explored by analysts, engineers and so-called "data scientists" in commercial industries and private institutions and areas that include information technology management, marketing, finance pharmaceutical and medical care.

Sam Madden, an associate professor at MIT who is also bigdata@CSAIL’s leader will be working to coordinate activities, connect partners and academics to create and feed projects.

“Though we’re hearing hype around the big data term, data has been accumulating for a long time,” says Madden. “I think of big data as a longer term movement to extract much more sophisticated inferences from large data sets, ones that are much less clean and are noisy compared to the data that exists inside traditional relational databases.”

Benefits are manifold, Madden says, and can extend from security research to commercial understanding of customer information to in-depth processing of medical records that could result in better diagnoses and treatment for patients.

“You’ve got sensor data coming in from cars and phones to use to understand traffic patterns, long histories of financial transactions that can predict default, genomes of people and cancers that will help you understand the person in your clinic,” Madden says. “We are learning with more and more analytics and people are seeing that machine learning really does work. Now, we want to apply that at scale and it will be a challenge.”  

Massachusetts Governor Deval Patrick also attended the rollout and announced a new statewide initiative to make Massachusetts a hub and a “premier destination” for big data research.

The ISTC center being funded by Intel will provide $2.5 million per year for up to five years to the center, which is being headed up by Madden and data management visionary Michael Stonebreaker, who is also on staff at MIT and CSAIL. 

Founding members of bigdata@CSAIL include industry partners AIG, EMC, Intel, SAP and Thompson Reuters. More partner are being set and an advisory board has been established to set the agenda.

Madden says much of the organizing is just under way and more partners, up to perhaps 10, are being sought. More information is available at the group’s website www.bigdata.csail.mit.edu.