June 6, 2012 – IBM, for instance, estimates that 2.5 quintillion bytes of data — one billion billion chunks – get created every day. And it’s no longer just financial statements and text documents. Its data from sensors used to gather climate information, videos, photos, social media posts and the like.

What is said in the Twittersphere is increasingly relied on for forming trading strategies and, in the most extreme case, taking real-time action on exchanges.

But the explosion of what is called unstructured or semi-structured data along with the traditional “structured data” like prices, share amounts and the like – as well as communications about all this data – can’t continue. Or at least, market participants’ ability to store everything that comes across their trading desks and back offices.

"Are we going to save everything? Right now, I think we probably are. But it's not sustainable,’’ said Mark Temple-Raston, senior vice president of the Office of the Chief Technology Officer at Citigroup.

Temple-Raston was speaking at the Market Tech 2012 forum of the Tabb Grop at the Times Center in New York. Where Daryan Dehghanpisheh, Global Sales Director, Financial Services Group, Intel, said the pace of growth in the amount of data to be processed is not going to let up.

"To bet that the treadmill is going to slow down or the incline is going to come down, it's just not going to happen,'' Dehghanpisheh said.

The surge is reflected in the amount of hardware and software that market researcher IDC expects will be sold to handle it. Last month, IDC forecast that the market for technology and services to deal with the growth in video, social media, market data visualization and the like is expected to grow from $3.2 billion in 2010 to $16.9 billion in 2015.

This, IDC said, is an annual growth rate of 40% -- or about seven times that of information technology in general.

Already in 2012, the Options Price Reporting Authority has seen a peak of 4.1 million messages a second and is apparently preparing to handle as many as 24.4 billion in one day – some time next year.

Much of what is driving retention of data is regulation coming out the Securities and Exchange Commission, the Financial Industry Regulatory Authority and the Dodd-Frank Wall Street Reform Act of 2010.

The question, according to Mark King, global strategy director for securities industry, for computer maker Dell, is not just how to keep the data, bug where and for how long.

For Citigroup’s Temple-Raston, it’s “how much of this incredible flow of data are we going to keep.’’

For failover purposes, a market player may only need a month or two of past data. Keeping everything is an “extravagance you don’t really need.’’

One way to determine what to keep and what not, when so much data is flowing across all markets and being kept more than once, is: who saw it first?, suggested Dehghanpisheh.

Who saw it first. Who acted on it first. Who redistributed it first. It will come down to policy and procedure, said Temple-Raston.

This story originally appeared at Securities Technology Monitor.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access