Into the Cloud, To Power Trading

Register now

The conventional wisdom has been that only back-office and administrative systems could be put into the "cloud" of computing, where capacity on networks and servers are shared-and your proprietary code is out of your hands.

Putting trading code in the cloud also is supposed to mean delay-which, if your strategies rely on speed, means you risk losing your edge.

But some smaller hedge funds and proprietary traders are using Amazon EC2's compute cloud to specifically power their trading algorithms, execute their automated strategies and store the results. They can also find help with supercomputing power, from academia ("Universities Put 'Blue Gene' Machines in Cloud, to Help Hedge Funds Trade").

Ernie Chan, a quantitative trader, consultant and co-founder of EXP Capital Management in Chicago, wrote a book on the business ("Quantitative Trading: How to Build Your Own Algorithmic Trading Business," John Wiley & Sons, 2008). Now he's using EC2 for his own trading.

"I use EC2 because it's extremely affordable," Chan said. "And it's much more stable than running everything on your own desktop, which I did for a couple of years. Co-locating your own servers at a broker can be quite costly-a few thousand dollars a month, which can be too much for a startup hedge fund." EXP pays about $100 per month to EC2 to power its trading.

Whiz kid Max Dama, a third-year student at the University of California at Berkeley who teaches an experimental class on quantitative trading, is known for applying artificial intelligence to finance at Berkeley's Center for Innovative Financial Technology. Dama's doing pairs trading using EC2 from his dorm: Algorithms place bets on the spread between two similar, typically correlated stocks, as one drops or rises against the other.

"[EC2] allowed me to self-start," Dama said. "If I were to attempt to support this on my own, I would have to hire a whole army of developers just to manage the servers."

While large funds and banks have generally dismissed public clouds as too insecure to entrust core operations such as trading, a new generation of technology savvy "quants"-those highly educated folks taken to applying mathematical and scientific methods to trading securities-are using public clouds such as EC2 to break out on their own.

At the very least, according to those doing it, public clouds represent an accessible and very cheap source to begin trading. For instance, it costs 68 cents per hour to run an extra-large, on-demand, compute-intensive processing program on EC2's cloud located in Northern Virginia, according to Amazon's EC2 website. That rate grants a user roughly 16 central processors (via 8 "virtual" dual cores), 7 gigabytes of working memory and 1,690 gigabytes of storage.

Such clouds can be viewed, therefore, as either a quick-launch option for startup funds or prop traders, or a stepping stone on which to perch until one garners enough disposable income to buy more expensive access, like co-locating servers with a broker.

The simple question many of these traders have asked themselves before hanging their own shingle: Why should I go through the rigors of trying to join a large firm and get noticed there when I can apply my ideas and strategies, which I know are proved and successful, right here in my own room and make money doing it now?

That, Chan said, represented his motivation: "I wanted to be my own boss."

Increasingly affordable and easy access to powerful resources like the EC2 cloud and electronic direct market execution services like that of Interactive Brokers have in effect spawned a second-generation "day trader." Essentially: the day trading quant.


Amazon began to rent unallocated capacity suitable for high-performance, compute-intensive applications in the giant data centers it had originally set up to fill online book, music and merchandise orders on Oct. 16, 2007. This enabled quants who wanted to trade for themselves, like Chan, to get in the game.

They got cheap processing power and storage. They could bet on their best algorithms, make money and take home all the returns (at least until they added investors). Call it a lone-wolf hunt for alpha. Or, high-speed hedge fund formation, part two.

"We use Amazon EC2 for some of our trading engines in our fund," said Chan, who began trading on EC2 in March 2009. EXP has $10 million under management.

"These programs are connected to market data feeds and will automatically submit orders to our brokerage account when appropriate,'' Chan said. "So far, we have not used EC2 for back-testing or research."

The latter two-testing trading strategies on historical data, or doing analyses, which don't involve immediate or near-term trading-are more typical of how public computing clouds are used in capital markets. But EXP actually places orders in the cloud. EXP's order management system sits on EC2.

"We use EC2 for actual order submission and automatic risk and position management, such as hedging, exiting, etc.," Chan said. "This is because the EC2 servers are in a more stable environment with respect to both computer and Internet connections than our own office computers and network can ever achieve, at a similar cost."

The setup does create about 1 microsecond to 1 millisecond of delay in getting EXP's orders to the matching engines of exchanges and other trading venues, which can be hundreds to thousands of miles from EC2's data centers. EC2 does not publish exact locales; its website says it has data centers in Northern Virginia, Northern California, Ireland and Singapore.

But the fact that Amazon's data centers are not located near exchanges doesn't matter to Chan, because "the strategies we run on EC2 are intraday; they are not high frequency," he said. The orders can be filled any time during a day's trading session, from 9:30 a.m. to 4 p.m. Chan trades from his house in Niagara-on-the-Lake, Ontario; his partner works from EXP's office in Chicago.

His automated trading system works as follows: Electronic feeds shoot market data into the logic that Chan and EXP have created to decide whether to submit an order. When a buy or sell order is triggered, the details are sent to the transaction interface of a broker-dealer it uses-which also sits on a server in the EC2 "cloud."

The same program that enters trades will also manage existing orders, deciding whether an order should be canceled and replaced, or if a position should be exited or modified.

EXP's entire order, decision and risk management are conducted using one big program, running continuously on the EC2 server.

Chan's expertise is in statistical pattern recognition. Like many well-known quants who have moved into finance, Chan got his start in IBM's Human Language Technologies group, which specializes in voice recognition.

Many high-profile hires have come from this IBM group, as well as from other rarefied disciplines such as cryptology, physics and mathematics. These include Peter Brown and Robert Mercer, who became co-CEOs at Renaissance Technologies the first of last year after founder James Simons stepped down from day-to-day management. Since its inception in 1988, Renaissance's well-known Medallion fund has averaged 45 percent returns, net of fees, every year, according to The Wall Street Journal.

Chan has developed statistical arbitrage trading models for hedge funds, including Millennium Partners, Mapleridge Capital Management, MANE Fund Management and for the prop desk of Credit Suisse. He also spearheaded text-based data searches involving statistical algorithms at Morgan Stanley, aimed at uncovering theretofore untapped relationships among the bank's client accounts.

Chan says, however, that public clouds are no panacea for low-cost automated trading. Some users, including consultants and cloud server monitoring firms AW2.0, Cloudkick and enStratus, have said Amazon EC2 suffers from slowdowns during peak use. Cloudkick has said the latency could have more to do with the hardware configurations of EC2's customers, versus the service lacking sufficient capacity. But outages do occur. The most recent involved a May 11 car crash into a utility pole, which took down service to certain users in the East Coast region, according to

Amazon said at the time it was able to get the service back up and running quickly. Similar outages affect many providers, with causes ranging from squirrels chewing wires to an errantly handled backhoe.

Using EC2 for trading may also supply diminishing returns as one's business grows. That's at least the case for EXP, which now has made enough money to move off EC2 and take advantage of faster infrastructure with more reliability. At press time, Chan said he was in the process of moving his trading system to servers co-located with his broker in Jersey City, N.J., less than 8 miles from each of the New York Stock Exchange's nearest three data centers in Lower Manhattan, downtown Brooklyn and Weehawken N.J. (Brokers in those three facilities are in the process of migrating their co-location equipment to NYSE Euronext's new Mahwah, N.J.-based data center, which is slated to power trading of NYSE stocks beginning in August.)

"I will be migrating from EC2 to brokerage co-location for all our trading servers to remove that last remaining Internet latency," Chan said.

"The reason is, even though the environment is stable at EC2-no complaints there-and it's up 24/7, both in terms of the network connection and work on the machine, it is nevertheless still subject to the vagaries of the Internet itself, which EC2 can do nothing about. If there's just a minor disruption in the Internet or network interference between the Amazon server and the brokerage, say something happens in between Seattle and New York, we are out of luck."

Short of Amazon getting into co-location and establishing a data center next to the major exchanges-and there have been no indications it will do that-there's nothing the online retailer can do to dissuade Chan from leaving the cloud to co-locate at EXP's executing broker, he said.

Amazon did not return multiple requests to comment on how it serves or plans to retain financial customers for its EC2 cloud services.

Chan emphasized, however, that "the advantage of using EC2 is when you're an individual trader or startup hedge fund that can't afford a couple thousand dollars a month for co-location. It's so cheap that any fund, regardless of the level of assets they manage, can use it. Amazon charges about $100 a month at the most, and that's when you're constantly logging on to the server."



Max Dama is pursuing a bachelor's degree in mathematics, statistics and business administration at Berkeley, with a particular focus on computer science, machine learning and quantitative trading.

In his spare time Dama teaches, manages a portfolio of his own money using automated trading on EC2, and does consulting work for students, other traders and financial system developers.

"I run a pairs strategy on EC2," he says. "I've used it to both run my own trading strategies and to run other peoples'. I also use it to manage and share the source code of my projects and to run large-scale statistical analysis."

To that end, he uses EC2 as if it were a cluster of computers, in his own data center.

"My system monitors trades in stocks that are known to be correlated or co-integrated, that whenever they deviate from certain ratios, it automatically executes a trade," Dama explained. "Whenever a market-maker changes a bid or ask price-every new incoming tick-that information goes directly to my system, which through an algorithm checks whether or not the ratio goes over a certain threshold. If it does, it immediately sends a signal to buy or sell." Results have been "inconsistent," he says, but he's still in "beta."

Dama says he hasn't had any problems with the continuity of service on EC2. "The connection to the Internet is really good-10 megabytes per second. That's the fastest you can get in the United States. It's not adjacent to the exchange. So it will be slower than co-locating, but it's much more cost-effective, particularly if you don't have an application that needs to shave off 100 microseconds. Then it's equivalent to professional-grade."

The benefit of using Amazon's infrastructure, in his experience so far, is "great uptime" and server administration. "I have three servers running-two Linux servers and one Windows server on EC2-and they've been going for about a year and a half now, and I've only spent $2,000 total for those," he said. "That's nonstop, continuous 24/7 uptime. I haven't had any problem with it. This is so much cheaper and much less of a hassle than if I was managing my own servers."

Dama hosts his source code on a third-party service, called Dropbox, which itself is hosted on EC2. He "drops" files into the box, which can be used by partners collaborating on developing algorithms and strategies.

Dama has five other students at Berkeley who code his trading apps. He also rents programmers from India, Pakistan, Romania and Ukraine, whom he hires for $25 per hour over

"Some of the best developers in these countries are on these services," he said. Code-for-hire jobs are sometimes the best option available for both providers and users.

"People around the country or world log into that source box and make modifications to my code," Dama said. "If anybody does something wrong, I can revert back to previous versions, because every time you exchange data on EC2, it automatically backs up any changes. It took about 15 minutes to set that up and launch."

Dama chalks up wariness about the security and stability of services like EC2 among financial firms to corporate tendencies to fear what they can't completely control in-house or that which seems creatively destructive and innovatively disruptive to traditional profit streams of outsourced or hosted services.

The conventional wisdom that public clouds are truly unworkable or inoperable in the enterprise says more about the corporation than the cloud, Dama says. "That's more about fear of the in-house software developers and security administrators not wanting to get fired."

Online, Amazon describes the service's privacy and security infrastructure as "world class." This is expected from one of the globe's largest online retailers responsible for millions of consumers' financial details, a claim that may prove insufficient for some firms. One option for firms needing more audited safeguarding is to tap third-party identity management services designed specifically to secure customer data and access on EC2, like those of Boulder, Colo.-based Symplified, which offers the Symplified Trust Cloud, built to bolster defenses on EC2 to ensure customers are compliant with enterprise-level security standards and regulations when using the cloud.



Several consultants in the blogosphere as well as those contacted by Securities Industry News claim a number of funds use public clouds for services that either support or include trading, but they're doing it under the radar, so as not to freak out management or investors. For instance, a couple hedge funds are using University-owned hardware in a program sponsored by New York State to develop algorithmic trading systems (see sidebar).

One technology director and programmer at a top buy-side investment house with nearly $180 billion in assets under management said many financial firms may not realize their third-party apps are hosted on EC2 already.

Fidessa, for instance, uses EC2 to host its Fragulator, which enables users to query "a billion trade records" to get a picture of how a given stock trades across exchanges, dark pools and bilaterally in over-the-counter markets.

Tacoma, Wa.-based Russell Investments, creator of the well-known Russell indexes, is open about using Amazon's EC2 cloud to store and access time series data from its analytics vendor to enhance its fixed income offerings. There is no trading, and no running of analytics packages. But Russell's fixed income group uses the cloud-hosted data on the Web to build historical trend analysis used in trade reporting, says Randy Kelley, senior solutions architect at Russell.

The reports go to senior and executive management in the case of market events, and to determine the funds' sector and stock exposure. Research analysts also run these reports and discuss the results with Russell's fund managers.

"Russell downloads the archived files, which are extraordinarily large and often resource-intensive, from Amazon's cloud," said Kelley, who has promoted use of EC2 as well as's cloud offering,, at Russell. "EC2 provides a fairly inexpensive storage option. It also allows Web delivery of the analytics to our fixed income team. When historical data is needed, fixed income analysts can tap into the data from the cloud."

 This story first appeared in Securities Industry News.

For reprint and licensing requests for this article, click here.