In 2,500 square feet of space, you can house a typical Radio Shack store, with hundreds of electronic products, ranging from smart phones to car security systems to MP3 players to weather radios to cables and batteries.

Or, in roughly the same amount of space, you can house a stock exchange. Or two.

Such is the case with EDGA and EDGX, the two newly approved stock exchanges owned by Direct Edge, which formally debut July 21 in Secaucus, N.J. As competition in securities trading venues goes up, so does the drive for speed and “scalability,’’ the ability to expand rapidly and reliably.

Because Direct Edge, like other venue operators before it, takes its approval to operate full-fledged exchanges as a sign that it is here to stay, competing on the same fields as the New York Stock Exchange, Nasdaq Stock Market and BATS Exchange. And, concomitant with that, Direct Edge wants to prove that it can.

“What Direct Edge is trying to do is build a company that can be an enduring part of the exchange community for a long time,’’ said chairman and chief executive William O’Brien. “And that transition to exchange status is part of that evolution.’’

The economic benefits (see “Benefits, and Costs, in the Life of an Exchange”) and the prestige may be big factors. And the transformation of its alternative venues into full-fledged exchanges and the technology transformation that has gone with it may just be the price of keeping up with rivals.

“It’s a hypercompetitive environment,’’ said Sang Lee, managing partner of the Aite Group, a Boston industry consulting firm. “You’re literally gouging each other’s eyes out for a sort of single-digit market share” in the Unite States.

The changeover of its venues into exchanges and the near-complete makeover of its technology are necessary steps for Direct Edge “ to continue to grow,’’ Lee said on the eve of the launch of both.

But Direct Edge’s stage is not large. It does not have to be.

NO FLAG

Unlike the New York Stock Exchange, you won’t see a monster-sized flag draped over where Direct Edge operates the new trading platforms that will power EDGA and EDGX. The two exchanges sit side by side, in a black wire cage inside a nondescript industrial building that could just as easily hide a distribution warehouse.

Inside the cage are 80 racks of Hewlett-Packard ProLiant servers and related memory storage units. Anywhere from two to 14 servers sit in each rack. About 200 servers, per exchange. By H-P’s estimate, this generation of “convergent” servers – which include processing capacity, memory storage, networking features and energy efficiency – puts the same amount of computing power in a single box as was found in 21 boxes four years ago.

Spare servers in this cage provide immediate backup, in case of failure. A second cage with copies of the entire EDGA and EDGX trading exchanges sits in another nondescript building in North New Jersey, just in case.

It’s a major transformation for a company which started a dozen years ago as an electronic communications network known as Attain. Back then, Attain was trading just 1 million shares a day. “Nobody paid any attention to it,’’ said Lee, when it was owned by All-Tech Investment Group. In fact, the most notoriety came when a day trader went on a shooting spree at All-Tech and one other firm in July 1999, leaving nine dead.

Now Direct Edge is owned by the International Securities Exchange, Knight Capital Group, Citadel Derivatives Group and The Goldman Sachs Group. The company “has literally outpassed everyone that was ahead of them, in two or three years,’’ said Lee. “To me, that’s impressive.

The new platform developed, constructed and installed over the last 18 months replaces a single data center, which had topped out in its ability to handle data about market conditions and transactions at 60,000 messages a second.

The new setup is a Windows-based system based on Intel Xeon 5500 processors, also known as Nehalem, announced last year. Where Direct Edge’s servers before processed 32 bits of data a time in a single thread of information at a time, now each processor handles 64 bits of data at a time, in multiple threads. Each Xeon processor, in fact, has multiple “core” processors on it. Each has two electric sockets. Each of those sockets accepts a circuit with four “core” processors on it.

So each server in Direct Edge’s racks actually has eight computers built in. And each of those processors can handle two threads of logic at once.

Which, in the case of EDGA and EDGX, now means that each order that comes in can be assigned its own thread.

This may seem simple and relatively straightforward. But in the push for speed and reliability, it makes it also possible to create systems that achieve both at the same time, in other simple ways.

DOUBLING DOWN

For instance, each order that comes in gets published twice. In effect, two copies of the order get sent to one of the exchange’s matching engines, according to chief technical officer Steve Bonanno.

That does not mean that the order gets completed twice. Instead, each order that comes in is sent in the same exact sequence to a pair of matching engines running on different servers. Each engine processes every order in a given set of symbols. But when the first message that an order has been processed is sent back to a subscriber, the other is discarded.

That means that if one copy of the order gets hung up, it doesn’t matter. The order still will go through, without having to identify it and resend it.

From the get-go, the new system will be able to handle 220,000 messages every second, while keeping up the pace of acknowledging each order in less than half a million. Each of the two exchanges can handle 2.5 billion transactions a day. “That is four times the capacity of even the days surrounding the Flash Crash,’’ said O’Brien. “So there is significant head room there.”

Capacity, in the future, can be expanded beyond that. Besides, as a point of reference, last Friday, the two exchanges handled trading in 782.8 million shares of stock, or about 400 million transactions. Together.

The prior Direct Edge system was not set up well to handle “microbursts,’’ said Bonanno. Those are the huge spikes in trading volume, when market participants get in a frenzy or their automated trading systems start creating unexpected interactions like that of the Flash Crash of May 6 to which O’Brien refers.

Now, the EDGA and EDGX systems will routinely acknowledge orders in about 340 millionths of a second. That is a gain of a roughly 75 percent, from 1.2 milliseconds. Among the key techniques: using solid-state memory, which never loses data, en route to the processors. This alone saves between 70 and 100 microseconds on each order. Another saving: Fast queueing of orders in buffers that feed each thread, with each buffer disappearing almost at the moment it’s created. And finally, Direct Edge has created its own interface to customers’ trading programs. This shaves a few microseconds off using the more widely used Financial Information Exchange protocol, in communicating and processing requests.

But speed is not everything. BATS and Nasdaq, for instance, already claim to process orders in 250 microseconds, on average.

“We have four major execution venues now in the public markets – NYSE, Nasdaq, BATS, and Direct Edge,’’ said Aite Group’s Lee. “ I don’t expect that to change dramatically because someone was able to shave off an additional 100 microseconds at this point.’’

Now, the bigger time savings may come from cutting the time spent sending orders between a customer’s site and the execution venue – and worrying about “distance itself,’’ said Lee.

“Now, something that shattered that mark, maybe 500 nanoseconds would make a difference to latency-sensitive folks out there,’’ he said of the time it takes for an exchange to process an order. “Even then, I don’t know you’d even notice the difference.”

GETTING THE MESSAGE

The answer for handling microbursts already was in house. The answer came when Direct Edge started to plan for the switchover of its alternative trading systems, EDGA and EDGX, into full-fledged national stock exchanges.

That’s when Direct Edge acquired the ISE Stock Exchange from the International Securities Exchange, in December 2008.

The ISE Stock Exchange already had installed specialized software, designed to filter and distribute large amounts of messages to various trading applications.

The software, known as Ultra Messaging for the Enterprise, came from a Chicago-based technology supplier, 29West. Its software created a backbone that allowed customers’ ordering systems, Direct Edge’s order-matching engines and its back-end compliance systems to talk to each other. When a particular action takes place – like the completion of an order – the results are published automatically to all interested parties and all necessary systems.

Also key was the use of the solid-state memory, from Texas Memory, which creates “persistence.” This means that even in the event of a power outage or other interruption, the information about trades does not disappear. When the system is restored, the data is intact.

“If we have a failure and we need to recover on another machine, the persistence is what allows us to recover,’’ John Ryan, Direct Edge’s chief system architect, told Securities Technology Monitor earlier this year.

The specialized system runs on the Hewlett-Packard servers and Cisco Systems networking gear already in Direct Edge’s cage at the Equinix NY4 data center in Secaucus, N.J. This allowed Direct Edge to cut out the need to install large numbers of messaging servers and additional networking gear now and as messaging volume expands.

And with Hewlett-Packard’s “convergent” technology, essentially all computing power, storage capacity and networking bandwidth goes into one giant pool of resources. Computing and communication bandwidth is broken into “virtual” machines. And if one virtual machine goes down, it can get isolated, notes H-P’s Jim Ganthier, marketing vice president for industry standard servers. Other processes continue uninterrupted.

THE NEXT POP

Not to get too far ahead of the game, but Bonanno is already looking forward to more speed and capacity, from the next generation of Xeon processors, the 5600 line introduced in March and known as Westmere.

These new processors can popped into place in the H-P ProLiant servers inside the Direct Edge cage, if need be. They’ll fit into existing sockets. And, once inserted, the capacity of each server goes up 50 percent, according to Intel.

When changed over, Bonanno intends to put order-entry engines onto the same chip as copies of the exchanges’ matching engines. That’s about as close together as you can “co-locate” order and matching engines. Even if customers’ connections to your engines are elsewhere in the cage. And even if the details of the transactions are cleared and settled elsewhere.

At that point, Bonanno expects to be able to get another 75 percent gain from Westmere – pushing acknowledgement times down to about 100 millionths of a second, the next speed target for major venues, as Aite Group’s Lee notes.

“There is no theoretical limit” to the amount of transactions Direct Edge will be able to handle each day, Bonanno said, at these speeds.

The Westmere chips are already available. Intel is just waiting, it says, until its customer calls and asks for the speed and capacity gains to be popped in.

The only question is how soon that call will come. “You don’t just sit back,’’ said Lee. “No one has the luxury of doing that these days.”

Direct Edge says it won’t wait long. “The migration to the next version of our technology effectively starts the same day as the launch,’’ O’Brien said earlier this month.

 

DIRECT EDGE AT A GLANCE

Founded: 1998

Owners: International Securities Exchange, Knight Capital Group, Citadel Derivatives Group and The Goldman Sachs Group

Headquarters: Jersey City, N.J.

Chief Executive Officer: William O'Brien

Chief Operating Officer: William Karsh

Chief Technology Officer: Steve Bonanno

Business: Operator of EDGA and EDGX national stock exchanges

  •  EDGA: Fees for providing or taking liquidity.
  • EDGX: “Takers’’ pay fees. “Makers” of liquidity get credits. Appeals to traders placing limit orders.
  • Volume: 1.17 billion shares a day, combined

Handles: 13.3 percent of shares traded in NYSE-listed securities, 12.5 percent of Nasdaq-listed securities

Technology: Windows- and Intel-based processing of industry standard and custom-written trading and messaging applications. Intel Xeon 5500 processors, Hewlett-Packard ProLiant G6 servers, Cisco Systems networking, Texas Memory RAM SAN solid-state memory.

Objectives: Speed, reliability, capacity, throughput

Measures:

  • Acknowledges an order in 340 millionths of a second
  • Can handle 220,000 messages every second
  • Ability to execute 2.5 billion transactions a day, per exchange

 
 

In 2,500 square feet of space, you can house a typical Radio Shack store, with hundreds of electronic products, ranging from smart phones to car security systems to MP3 players to weather radios to cables and batteries.

Or, in roughly the same amount of space, you can house a stock exchange. Or two.

Such is the case with EDGA and EDGX, the two newly approved stock exchanges owned by Direct Edge, which formally debut July 21 in Secaucus, N.J. As competition in securities trading venues goes up, so does the drive for speed and “scalability,’’ the ability to expand rapidly and reliably.

Because Direct Edge, like other venue operators before it, takes its approval to operate full-fledged exchanges as a sign that it is here to stay, competing on the same fields as the New York Stock Exchange, Nasdaq Stock Market and BATS Exchange. And, concomitant with that, Direct Edge wants to prove that it can.

“What Direct Edge is trying to do is build a company that can be an enduring part of the exchange community for a long time,’’ said chairman and chief executive William O’Brien. “And that transition to exchange status is part of that evolution.’’

The economic benefits (see “Benefits, and Costs, in the Life of an Exchange”) and the prestige may be big factors. And the transformation of its alternative venues into full-fledged exchanges and the technology transformation that has gone with it may just be the price of keeping up with rivals.

http://www.securitiestechnologymonitor.com/news/-25835-1.html

“It’s a hypercompetitive environment,’’ said Sang Lee, managing partner of the Aite Group, a Boston industry consulting firm. “You’re literally gouging each other’s eyes out for a sort of single-digit market share” in the Unite States.

The changeover of its venues into exchanges and the near-complete makeover of its technology are necessary steps for Direct Edge “ to continue to grow,’’ Lee said on the eve of the launch of both.

But Direct Edge’s stage is not large. It does not have to be.

NO FLAG

Unlike the New York Stock Exchange, you won’t see a monster-sized flag draped over where Direct Edge operates the new trading platforms that will power EDGA and EDGX. The two exchanges sit side by side, in a black wire cage inside a nondescript industrial building that could just as easily hide a distribution warehouse.

Inside the cage are 80 racks of Hewlett-Packard ProLiant servers and related memory storage units. Anywhere from two to 14 servers sit in each rack. About 200 servers, per exchange. By H-P’s estimate, this generation of “convergent” servers – which include processing capacity, memory storage, networking features and energy efficiency – puts the same amount of computing power in a single box as was found in 21 boxes four years ago.

Spare servers in this cage provide immediate backup, in case of failure. A second cage with copies of the entire EDGA and EDGX trading exchanges sits in another nondescript building in North New Jersey, just in case.

It’s a major transformation for a company which started a dozen years ago as an electronic communications network known as Attain. Back then, Attain was trading just 1 million shares a day. “Nobody paid any attention to it,’’ said Lee, when it was owned by All-Tech Investment Group. In fact, the most notoriety came when a day trader went on a shooting spree at All-Tech and one other firm in July 1999, leaving nine dead.

Now Direct Edge is owned by the International Securities Exchange, Knight Capital Group, Citadel Derivatives Group and The Goldman Sachs Group. The company “has literally outpassed everyone that was ahead of them, in two or three years,’’ said Lee. “To me, that’s impressive.

The new platform developed, constructed and installed over the last 18 months replaces a single data center, which had topped out in its ability to handle data about market conditions and transactions at 60,000 messages a second.

The new setup is a Windows-based system based on Intel Xeon 5500 processors, also known as Nehalem, announced last year. Where Direct Edge’s servers before processed 32 bits of data a time in a single thread of information at a time, now each processor handles 64 bits of data at a time, in multiple threads. Each Xeon processor, in fact, has multiple “core” processors on it. Each has two electric sockets. Each of those sockets accepts a circuit with four “core” processors on it.

So each server in Direct Edge’s racks actually has eight computers built in. And each of those processors can handle two threads of logic at once.

Which, in the case of EDGA and EDGX, now means that each order that comes in can be assigned its own thread.

This may seem simple and relatively straightforward. But in the push for speed and reliability, it makes it also possible to create systems that achieve both at the same time, in other simple ways.

DOUBLING DOWN

For instance, each order that comes in gets published twice. In effect, two copies of the order get sent to one of the exchange’s matching engines, according to chief technical officer Steve Bonanno.

That does not mean that the order gets completed twice. Instead, each order that comes in is sent in the same exact sequence to a pair of matching engines running on different servers. Each engine processes every order in a given set of symbols. But when the first message that an order has been processed is sent back to a subscriber, the other is discarded.

That means that if one copy of the order gets hung up, it doesn’t matter. The order still will go through, without having to identify it and resend it.

From the get-go, the new system will be able to handle 220,000 messages every second, while keeping up the pace of acknowledging each order in less than half a million. Each of the two exchanges can handle 2.5 billion transactions a day. “That is four times the capacity of even the days surrounding the Flash Crash,’’ said O’Brien. “So there is significant head room there.”

Capacity, in the future, can be expanded beyond that. Besides, as a point of reference, last Friday, the two exchanges handled trading in 782.8 million shares of stock, or about 400 million transactions. Together.

The prior Direct Edge system was not set up well to handle “microbursts,’’ said Bonanno. Those are the huge spikes in trading volume, when market participants get in a frenzy or their automated trading systems start creating unexpected interactions like that of the Flash Crash of May 6 to which O’Brien refers.

Now, the EDGA and EDGX systems will routinely acknowledge orders in about 340 millionths of a second. That is a gain of a roughly 75 percent, from 1.2 milliseconds. Among the key techniques: using solid-state memory, which never loses data, en route to the processors. This alone saves between 70 and 100 microseconds on each order. Another saving: Fast queueing of orders in buffers that feed each thread, with each buffer disappearing almost at the moment it’s created. And finally, Direct Edge has created its own interface to customers’ trading programs. This shaves a few microseconds off using the more widely used Financial Information Exchange protocol, in communicating and processing requests.

But speed is not everything. BATS and Nasdaq, for instance, already claim to process orders in 250 microseconds, on average.

“We have four major execution venues now in the public markets – NYSE, Nasdaq, BATS, and Direct Edge,’’ said Aite Group’s Lee. “ I don’t expect that to change dramatically because someone was able to shave off an additional 100 microseconds at this point.’’

Now, the bigger time savings may come from cutting the time spent sending orders between a customer’s site and the execution venue – and worrying about “distance itself,’’ said Lee.

“Now, something that shattered that mark, maybe 500 nanoseconds would make a difference to latency-sensitive folks out there,’’ he said of the time it takes for an exchange to process an order. “Even then, I don’t know you’d even notice the difference.”

GETTING THE MESSAGE

The answer for handling microbursts already was in house. The answer came when Direct Edge started to plan for the switchover of its alternative trading systems, EDGA and EDGX, into full-fledged national stock exchanges.

That’s when Direct Edge acquired the ISE Stock Exchange from the International Securities Exchange, in December 2008.

The ISE Stock Exchange already had installed specialized software, designed to filter and distribute large amounts of messages to various trading applications.

The software, known as Ultra Messaging for the Enterprise, came from a Chicago-based technology supplier, 29West. Its software created a backbone that allowed customers’ ordering systems, Direct Edge’s order-matching engines and its back-end compliance systems to talk to each other. When a particular action takes place – like the completion of an order – the results are published automatically to all interested parties and all necessary systems.

Also key was the use of the solid-state memory, from Texas Memory, which creates “persistence.” This means that even in the event of a power outage or other interruption, the information about trades does not disappear. When the system is restored, the data is intact.

“If we have a failure and we need to recover on another machine, the persistence is what allows us to recover,’’ John Ryan, Direct Edge’s chief system architect, told Securities Technology Monitor earlier this year.

The specialized system runs on the Hewlett-Packard servers and Cisco Systems networking gear already in Direct Edge’s cage at the Equinix NY4 data center in Secaucus, N.J. This allowed Direct Edge to cut out the need to install large numbers of messaging servers and additional networking gear now and as messaging volume expands.

And with Hewlett-Packard’s “convergent” technology, essentially all computing power, storage capacity and networking bandwidth goes into one giant pool of resources. Computing and communication bandwidth is broken into “virtual” machines. And if one virtual machine goes down, it can get isolated, notes H-P’s Jim Ganthier, marketing vice president for industry standard servers. Other processes continue uninterrupted.

THE NEXT POP

Not to get too far ahead of the game, but Bonanno is already looking forward to more speed and capacity, from the next generation of Xeon processors, the 5600 line introduced in March and known as Westmere.

These new processors can popped into place in the H-P ProLiant servers inside the Direct Edge cage, if need be. They’ll fit into existing sockets. And, once inserted, the capacity of each server goes up 50 percent, according to Intel.

When changed over, Bonanno intends to put order-entry engines onto the same chip as copies of the exchanges’ matching engines. That’s about as close together as you can “co-locate” order and matching engines. Even if customers’ connections to your engines are elsewhere in the cage. And even if the details of the transactions are cleared and settled elsewhere.

At that point, Bonanno expects to be able to get another 75 percent gain from Westmere – pushing acknowledgement times down to about 100 millionths of a second, the next speed target for major venues, as Aite Group’s Lee notes.

“There is no theoretical limit” to the amount of transactions Direct Edge will be able to handle each day, Bonanno said, at these speeds.

The Westmere chips are already available. Intel is just waiting, it says, until its customer calls and asks for the speed and capacity gains to be popped in.

The only question is how soon that call will come. “You don’t just sit back,’’ said Lee. “No one has the luxury of doing that these days.”

Direct Edge says it won’t wait long. “The migration to the next version of our technology effectively starts the same day as the launch,’’ O’Brien said earlier this month.

 

DIRECT EDGE AT A GLANCE

Founded: 1998

Owners: International Securities Exchange, Knight Capital Group, Citadel Derivatives Group and The Goldman Sachs Group

Headquarters: Jersey City, N.J.

Chief Executive Officer: William O'Brien

Chief Operating Officer: William Karsh

Chief Technology Officer: Steve Bonanno

Business: Operator of EDGA and EDGX national stock exchanges

  •  EDGA: Fees for providing or taking liquidity.
  • EDGX: “Takers’’ pay fees. “Makers” of liquidity get credits. Appeals to traders placing limit orders.
  • Volume: 1.17 billion shares a day, combined

Handles: 13.3 percent of shares traded in NYSE-listed securities, 12.5 percent of Nasdaq-listed securities

Technology: Windows- and Intel-based processing of industry standard and custom-written trading and messaging applications. Intel Xeon 5500 processors, Hewlett-Packard ProLiant G6 servers, Cisco Systems networking, Texas Memory RAM SAN solid-state memory.

Objectives: Speed, reliability, capacity, throughput

Measures:

  • Acknowledges an order in 340 millionths of a second
  • Can handle 220,000 messages every second
  • Ability to execute 2.5 billion transactions a day, per exchange

 
 

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access