High-Performance Computing Makes a Splash in Water Cycle Science

The San Diego Supercomputer Center (SDSC) and the Scripps Institution of Oceanography (Scripps), both located at UC San Diego, have forded a stream between high-performance computing (HPC) and water cycle science.

This summer, SDSC’s petascale Comet supercomputer—which can perform nearly 3 quadrillion operations per second—will conclude formal service as a National Science Foundation (NSF) resource and transition to exclusive use by Scripps’ Center for Western Weather and Water Extremes (CW3E). The transition enables CW3E researchers to leverage Comet’s computing capabilities to improve weather and hydrological forecasts with the goal of enhancing the decision-making process associated with reservoir management over California, which could result in increased water supply and reduced flood risk over the region.

SDSC has been working to transition other Comet users onto Expanse, SDSC’s newest NSF-funded system, which entered production operations in December of last year. “Comet has been a workhorse system for the NSF, and we are thrilled with the opportunity to extend its useful life in support of this important work,” said Michael Norman, director of SDSC and principal investigator for Comet.

According to Luca Delle Monache, academic program manager of climate, atmospheric science and physical oceanography at Scripps, the exclusive access to Comet will allow CW3E to advance several computational projects that would not be possible otherwise.

“The ability to perform computational experiments that just one year ago we couldn’t even think of and to strengthen our collaboration with key partners as the National Weather Service (NWS) is an exciting aspect of this agreement,” he said.

Delle Monache explained that CW3E’s computational projects will span a broad scope from basic to applied research. The projects will range from trying to answer fundamental questions on the formation and evolution of atmospheric rivers, and the interactions between the atmosphere and the ocean, to studying orographic precipitation, which is generated when the moist air mass associated with an atmospheric river is lifted over mountains and condenses into rain or snow. The projects will also include developing high-resolution, subseasonal to seasonal predictions; testing and improving operational systems run from the NWS; expanding CW3E near real-time operational capabilities for weather and hydrology with high resolution deterministic prediction, and an ensemble with an unprecedented large number of members.

“We will also develop and test new machine learning algorithms and dynamical models leveraging Comet’s several graphics processing unit (GPU) nodes,” said Delle Monache. “Moreover, we will develop and test new data assimilation schemes.”

Delle Monache noted that a CW3E Supercomputing Advisory Group (CSAG) has been established. The group includes leading experts from Scripps Institution of Oceanography, the National Center for Atmospheric Research, University of Arizona, University of Colorado at Boulder, Jet Propulsion Laboratory and the European Centre for Medium-Range Weather Forecasts.

“CSAG is helping us in designing some of the projects mentioned above. In addition to those, we are also exploring the possibility to dedicate substantial computational resources to students and universities from underrepresented groups, to allow for new or expand existing modeling experiments as part of their dissertation research,” said Delle Monache.

According to CW3E’s HPC Lead Pat Mulrooney, compared to the computational capabilities of other supercomputer resources, Comet provides access to 144 NVIDA GPUs, “not a common feature in the nation’s supercomputer clusters,” Mulrooney said. “Also, Comet has on node solid-state drives (SSDs) and the highest random access memory (RAM) to core processing units (CPU) core ratio.”

In late 2013, the NSF awarded SDSC an initial $12 million to build and operate Comet, which went into production in 2015. Since then the total NSF funding increased to about $27 million. Leading up to the end of its NSF tenure in July 2021, SDSC and CW3E worked jointly to develop the plan and timeline for supporting CW3E operations, including on-boarding of CW3E users and data.

“Over the last six years, Comet and the team at SDSC have provided a great resource to the science and engineering community, benefiting many researchers and educators across the nation and enabling insights and discoveries across all of science and engineering,” said the NSF’s Office of Advanced Cyberinfrastructure Director Manish Parashar. “NSF would like to express our appreciation to the SDSC team for their commitment to supporting the national research enterprise and inspiring new innovations in research cyberinfrastructure.”