The growth of electronic trading has altered the securities markets to such a degree that if a stock trader from a Wall Street firm fell asleep in 1998 and woke up today, he or she would have to relearn their job from the ground up. And one of the first things they would need to grasp would be the extreme volumes and bone-shaking velocity that define today’s financial markets.

Trading stocks (equities), bonds (fixed income) and foreign currencies (FX) has become a new lightning-speed world, one that gets a bit faster every day. Every month it seems there are fewer traders in exciting blazers waving their hands around on the floor of the stock exchange (much to the dismay of financial television presenters), while at the same time the total number of quotes sent electronically per second grows by leaps and bounds. There are far more transactions, and sometimes less profits: the introduction of decimalization - stocks being priced in cents, rather than 1/4, 1/8 or 1/16 parts of a dollar - is one factor that has led to a decrease in the margins for many brokers.

The growth of electronic trading means that more and more trades are being executed, sometimes in miniscule amounts. At the same time, analysis of historical trends and pricing relationships between securities is becoming ever more time-critical. The result is that each day stresses increase on hardware, network and data management systems. Every week the potential grows for a possible system breakdown, and every year Wall Street firms have to change their system architecture just to buy breathing room for a few more quarters of growth. For financial services professionals, the extreme has become routine.

A Newer, Faster World

The imposition of new financial regulations and the growth of technology over the past decade have pushed traders to take large orders and break them into many small parts. In 1997 the average number of shares per trade was almost 1,500, while last year it was just over 300.

As a result, there has been an avalanche of smaller and smaller trades, with the central system that handles all the stock quotes for the major exchanges growing at an incredible rate. On January 8, 2008, there were 4.6 billion messages sent per day on CTS, CQS, OPRA and NQDS, and 701,000 messages sent per second. By July 8, 2008, 6.1 billion messages are expected per day and 907,000 messages per second, based on research data from TABB Group, the financial markets research and strategic advisory firm, November 2007. In just six months, the number of quotes per second should grow by nearly 30 percent. This is an annual growth rate of almost 70 percent, and it may keep increasing.1

Trading isn’t the only business being transformed by newer, faster types of information sharing. Risk management, compliance and auditing functions are based in large part on examining market data to predict a company’s exposure to various types of risk. The more complex and dense data becomes, the more difficult accurate risk management becomes.

The Impact on Data Velocity

As trading systems move from being human-based to computer-based, decisions are getting made at far greater speeds. In the time it takes a trader to read a changing price on a computer screen, an automated trading system can evaluate a host of variables and execute a trade. So where trading was once a battle between brokers via their traders on the floor, it is evolving into a struggle of machine versus machine. And the machine that responds the quickest often gets the best deal.

This situation has led to a “low latency” arms race in which brokers push to make their system the fastest responder. To give some idea of the lengths that brokers are going to, some are renting space in data centers adjacent to exchanges so their machines are closest to the central trading systems. The speed of a broker’s network is becoming a critical, if not the critical factor in whether the broker gets the best deal.

The IT Challenge

Needless to say, IT departments at major brokerages are on the front lines of this revolution. IT officials need to make sure their systems can handle huge volumes of data (that keeps growing by the week) while at the same time ensure that the systems act on that data more quickly than their competitors.

Old technology is no longer up to the task. For example, storing the data to disk, then running queries simply will not work any more: the 900,000 messages per second have to be processed as they arrive. Frankly, traditional architectures cannot cope.

New technologies, such as complex event processing (CEP) engines, are emerging to handle the new world of electronic trading. A CEP engine can identify a preprogrammed pattern of data in the incoming message stream, and then send a signal to a trading system. This is the current state-of-the-art method of dealing with velocity.

Data storage and access is also changing. Instead of storing data in rows, as in a traditional databases, some products now store data in columns (Sybase’s IQ product has been using this approach for over 10 years, for example). This creates a more efficient use of disk space (compressing the original data in half, as opposed to expanding it by a factor of three, for example) and allows for much faster retrieval of the data – up to 100 times faster than an current architectures.

There is always a trade-off for this type of performance, and these column based databases are less effective in a traditional interactive system. However, for the financial services industry, the potential to gain greater data compression and increase retrieval speed outweighs any negatives.

The Future is Already Here

The revolution caused by the growth of electronic trading is irreversible, and the speed and density of data being generated will only increase each year. In the near future, more financial service applications will move to a new technology platform, using column databases to retrieve their historical market data and CEP engines to enable their systems to make instant decisions.

And while the financial services industry is on the forefront of that change, it will not be alone. Other industries will face similar challenges in the years ahead. For example, volumes of radio frequency identification (RFID) data are predicted to grow exponentially in the next few years, which is a major challenge for retailers and wholesalers. Very soon, the need to store vast amounts of data that also needs to be immediately accessible will be a challenge for most businesses, not only cutting-edge financial institutions.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access