NICS joins petaflop supercomputer club

The National Institute for Computational Sciences (NICS) brought its new petaflop-scale supercomputer online this month, joining a very rarefied club of systems that can reach a thousand trillion calculations per second.

The scientific researchers who will use this Cray XT5 system, which was funded through a National Science Foundation grant of $65 million, are working on “the most important problems in the world and they always gravitate to the fastest machines,” said Phil Andrews, the NCIS project director.

The machine will be used to study the formation of galaxies, climate change, biology and other sciences, running simulations and often in 3-D.

Kraken is an upgrade from a previous generation, 600 teraflop system, or one trillion floating point operations per second. As part of the upgrade, 16,000 six-core, 2.6-GHz chips by Advanced Micro Devices, were installed. It totals about 100,000 compute cores, spread across 88 cabinets. The previous configuration used AMD's 4 core Barcelona 2.3-GHz chip.

The NCIS system is housed at the Oak Ridge National Laboratory and is managed by the University of Tennessee. Planning for this system began several years ago and, at the time, Andrews said they had no real way of knowing just how fast this machine would be because of the uncertainties of chip clock speeds.

But, Andrews said, “We were hoping for a petaflop.”

IBM delivered the first publicly known petaflop system, the Roadrunner, a $100 million system, last year for Los Alamos National Laboratory.

Kraken is part of the TeraGrid, an NSF project that links high performance systems used in research, so getting research time on this machine isn't easy.

Proposals for compute time are reviewed by a peer review board that meets four times a year, and follow a process that's similar to submitting an article to a prestigious journal. Among the things researchers have to show before they can run their jobs is that their applications can scale to the system's capabilities.

Any supercomputer's useful life is roughly three to four years, said Andrews. “It's not like it is not useful after that, but it's no longer state of the art,” he said.

Kraken's speed allows greater resolution or finer analysis of scientific models. A slower machine you may be modeling the earth at 10 kilometer resolution, but a faster machine may take that resolution to as much as 5 kilometers. “That's important because some effects only manifest themselves at smaller resolutions,” said Andrews.

It won't be long before planning on a new machine begins. “The users are very unhappy if they don't think that there are going to be new machines coming along that will allow them to continue to develop their models — for them, this is their whole career,” Andrews said.