Overheard: AVATAR's Data Center
An Information Management interview with Paul Gunn, data center systems administrator, Weta Digital
Weta's data center isn't extremely large but is extremely fast for processing digital effects.
That's right, our business requires thousands of jobs and millions of tasks to make all of the 24 frames per second in a movie. Our center is about 10,000 square feet, and it's only half full. But we run a lot of machines - about 4,000 quad-core HP blades, 104 terabytes of RAM and three petabytes of network area storage. It all has to be packed together for performance, and we use multiple 10 gigabyte links to connect the gear.
Just the RAM you mentioned must mean you can do a lot of processing in memory.
Yes, we can. But we're not a supercomputer center or a chip design place where they have one huge job. We're very much locked into individual batch tasks, and that's why we have large numbers of machines. An artist creates an image, it gets reviewed, you add a spaceship, get the movement right - these are all iterative tasks in a job. For AVATAR, we ran 10,000 jobs per day and 1.3 or 1.4 million individual tasks. It's lots and lots of parallel processing units all doing discrete tasks.
So virtualization doesn't make sense for you?
For us it would only get in the way. Virtualization is great when you have resources to spare on a machine, whereas a job here will use all of a machine's CPU. Instead of virtualization, we use Pixar software called Alfred to create hierarchies of jobs and send them off to the machines in the order they're needed.
What are your issues running the network? Latency? Throughput?
The biggest issue we had coming up to AVATAR was the physical environment. In our new data center, the gear was getting very dense and fast, but also very hot and it was hard to even keep the rooms cool. Our old rooms were raised floors and air conditioning. That just didn't scale for the all the processors we run now.
What was the answer?
We went for water-cooling where the racks are enclosed and all the hot air runs through a chilled water radiator, just like the radiator on a car. The cooled air gets pumped through the front of the machine again. It was the only plan that worked, and it only rarely uses air conditioning.
How's your electric bill?
Less than you might think. Water is much denser than air and we capture heat in the racks so we don't have to cool the whole room. With our maritime climate, we mostly pay to run the water pumps. Plus, we run the machines warm, because the newer gear doesn't mind that. Keeping the machines at 25 degrees Celsius saves us many thousands of dollars a year. The engineers who designed it just won a national award for efficiency.
What keeps you awake at night?
Well, it's a very complex system so mostly we're glad to see the lights on. Handling the work on a project like AVATAR, you need to work very fast but not push your luck. We were processing seven or eight gigabytes of data per second for more than a solid month toward the end. If something had broken, it would have backed up a lot of people and that would not have been a pretty sight.
For more about Weta's data center, read "Processing AVATAR" at info-mgmt.com.