UCRL-CONF-211681
100+ TFlop Solidification Simulations on BlueGene/L
Frederick H. Streitz, James N. Glosli, Mehul V. Patel, Bor Chan, Robert K. Yates, Bronis R. de Supinski
Lawrence Livermore National Laboratory*, Livermore, CA 94550 USA
James Sexton, John A. Gunnels
IBM, Thomas J. Watson Research Center, Yorktown Heights, NY, USA
We investigate solidification in tantalum and uranium systems ranging in size from 64,000 to
524,288,000 atoms on the IBM BlueGene/L computer at LLNL. Using the newly developed ddcMD
code, we achieve performance rates as high as 103 TFlops, with a performance of 101.7 TFlop
sustained over a 7 hour run on 131,072 cpus. We demonstrate superb strong and weak scaling. Our
calculations are significant as they represent the first atomic-scale model of metal solidification to
proceed, without finite size effects, from spontaneous nucleation and growth of solid out of the liquid,
through the coalescence phase, and into the onset of coarsening. Thus, our simulations represent
the first step towards an atomistic model of nucleation and growth that can directly link atomistic
to mesoscopic length scales.
INTRODUCTION
Understanding the properties of matter under extreme
conditions is fundamental to researchers in fields as dis-
parate as astrophysics, planetary science and nuclear
physics. Scientists increasingly rely on computer mod-
els to develop this understanding since experiments are
often impossible (or extremely difficult) at pressures and
temperatures of interest.
Simulating complex systems (such as transition metals
or actinides) under extreme conditions poses several diffi-
culties. As the disorder in the system increases, describ-
ing the relevant physics requires increasingly large sys-
tems. Further, the length of time necessary to model non-
equilibrium behavior is also increasing. Systems with
hundreds of billions of atoms have been simulated but
these typically use computationally inexpensive pair (or
pair-like) potentials that are inadequate for the complex
systems that we are discussing. Billion atom simulations
are feasible using highly accurate many-body quantum-
based interaction potentials, such as MGPT[6–8]. Ulti-
mately, however, the required time scales to model non-
equilibrium behavior require a shorter wall-clock time to
solution than has been previously achieved. The new
generation of massively parallel computers, such as the
IBM BlueGene/L (BG/L) at Lawrence Livermore Na-
tional Laboratory, provide the computational power nec-
essary to achieve that improvement. This improvement
entails reducing the size of the problem on each proces-
sor, a severe test of the strong scaling limit for a code.
We developed a parallel classical molecular dynamics
(MD) code (ddcMD) that achieves the required strong
scaling and efficiently implements the MGPT potentials
on BG/L. Using ddcMD on BG/L, we have modeled the
pressure-induced solidification of molten tantalum and
investigated solidification in quenched uranium. Our
tantalum simulations have yielded the first determina-
tion of the minimum sized needed to model solidification
through the rapid nucleation and growth and phase, dur-
ing which individual grains are formed, to the coarsen-
ing phase, during which the metal solidifies into a char-
acteristic microstructure. The recently started uranium
simulations are producing valuable data and we eagerly
await seeing a solidification event as the system contin-
ues to run. In the first seven hours of the run, we have
already achieved the highest sustained performance mea-
sured with an applications code.
The rest of the paper is organized as follows: after a
brief overview of the BG/L architecture and the MGPT
interaction potentials, we describe the features of ddcMD
that enable simulation of complex systems under extreme
conditions (namely, a particle-based domain decomposi-
tion algorithm and a kernel of hand-tuned linear-algebra
routines), including a discussion of the performance and
scaling of the code on up to 131,072 processors of BG/L.
We then present our simulation results.
BLUEGENE/L ARCHITECTURE
BlueGene/L (BG/L) is a massively-parallel scientific
computing system developed by IBM in partnership
with the Advanced Simulation and Computing program
(ASC) of the US Department of Energy’s National Nu-
clear Security Agency [23, 24]. BG/L’s high-density cel-
lular design gives very high performance with low cost,
power and cooling requirements. The 65,536-node sys-
tem at LLNL is at this writing the fastest computer in
the world; the first half of the machine achieved 136.8
Tflops on the Linpack benchmark in May, 2005.
A compute node of BG/L is composed of only 10 chips:
its 700 MHz compute ASIC plus nine DRAM main mem-
ory chips. This highly integrated design drastically low-
ers power consumption and space requirements while fa-
voring communication and memory performance. The
BG/L ASIC has two independent PowerPC 440 cores,
each capable of two floating point operations per cycle
(c) 2005 Association for Computing Machinery. ACM acknowledges that this contribution was authored or
co-authored by a contractor or affiliate of the [U.S.] Government. As such, the Government retains a nonexclusive,
royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.
SC—05 November 12-18, 2005, Seattle, Washington, USA (c) 2005 ACM 1-59593-061-2/05/0011?$5.00