Argonne’s Aurora supercomputer and upgraded Advanced Photon Source will be powerful tools for discovery. Together, they’ll form a scientific supermerger: The combined data collection and computing power will advance discovery time and unlock new science.
Metals crack, neurons misfire, viruses mutate — all at scales of size and time we can barely fathom, let alone study.
To understand how processes work or fail in the natural or mechanical world, to solve the riddles of science, requires the ability to probe more deeply and expose layers of detail never before observed.
To gain that understanding — that knowing — requires access to powerful scientific tools.
Tools of such magnitude are used throughout the global scientific community. Some of them call the U.S. Department of Energy’s (DOE) Argonne National Laboratory home, and for decades, researchers from around the world have chosen Argonne as a place to conduct their work. Two of the main reasons are the supercomputing power of the Argonne Leadership Computing Facility (ALCF) and the high-energy X-ray beams generated at its Advanced Photon Source (APS), both DOE Office of Science user facilities.
“This combination of Aurora, the upgraded APS and AI will present opportunities in scientific exploration at speeds and scales previously inaccessible to science.” — Nicholas Schwarz, Scientific Software Engineering and Data Management group leader at the APS
But it seems the more we know about a subject, the more complex the questions become. So much so that even the technical capabilities at hand are no longer effective or efficient enough to pursue them.
To meet such challenges, Argonne is creating a formidable arsenal of tools through transformative upgrades at both the APS and ALCF.
Already renowned for its modeling, simulation, data analysis and artificial intelligence (AI) capabilities, the ALCF will introduce Aurora, a machine at the forefront of supercomputing, that will adapt new strategies in AI to more powerful hardware.
And the APS, currently one of the brightest hard X-ray synchrotrons in the Western hemisphere, will radically increase its performance to probe the inner workings of both animate and inanimate objects at unprecedented scales.
Remarkable as each machine will be on its own, a plan to integrate the two will create something of a single scientific mega-instrument designed to rapidly deliver analyzed data for on-the-fly adjustments to experiments and faster time to solutions.
To succeed, developers have to overcome a dilemma of their own making: the crush of big data. And the upgrades currently underway at both facilities will dramatically increase the amount of data they already produce and analyze.
Researchers estimate that experiments at the APS will collect hundreds of times more raw data from experiments after the upgrade than they do currently. In fact, the amount of new information will exceed all the songs in the entire Spotify catalog, for example. This increase in data is a result of the extraordinary speeds at which the upgraded APS will be able to capture snapshots of materials and processes and the phenomenal resolution of those snapshots. Scientists will be able to see much smaller things much more clearly and much more quickly.
Researchers are betting on Aurora’s advanced computational methods to aggressively digest that information, learn from it and represent it in a more concise and targeted form, and much faster than any assemblage of scientists ever could.
This tight coupling of APS experiment instruments with ALCF supercomputers will require an immense networking effort. In the future, the two facilities will be connected by dozens of fiber optic cables capable of transmitting trillions of data bits per second, allowing for hypercharged input of data and output of analysis.
Such technical supermergers could transform the way science is conducted, not only at Argonne but around the world. They could reduce time-to-discovery from years and months to weeks and days. But access to the machines, particularly on-demand access, could prove a stumbling block if not administered efficiently.
The ALCF will support on-demand data analysis by providing immediate access to Aurora to investigate topics that could have critical national importance, such as COVID-19. Keeping up with access demands is a complex problem with a lot of moving parts, but resolution could have implications beyond just Aurora and the APS.
“This is a new paradigm for the traditional supercomputing center,” said Bill Allcock, director of operations for the ALCF. “By supporting on-demand work, we’ll be capable of supporting new classes of science that, traditionally, have not been well-supported on large, high performance computing systems. There’s more science out there that can use this type of collaboration, and our goal is to support it all.”
Dynamic duo of technology
The APS acts like a powerful microscope that allows researchers to study complex materials and systems. Currently, the APS delivers X-ray beams that are up to a billion times brighter than those used by your dentist to penetrate deep inside materials. They are so bright, in fact, that scientists can use them to see, in real time, what happens inside a working battery or detail the atomic structure of a single protein.
The APS was built in the 1990s, and upgrading it into a state-of-the-art facility is a gargantuan endeavor. The original electron storage ring, two-thirds of a mile around, will be removed and replaced by a new, completely updated ring. Fueled by this powerhouse engine, the upgraded APS will generate X-ray beams that are up to 500 times brighter than those produced by the current machine, allowing scientists to examine materials and processes in greater detail and with greater precision than ever before.
With the upgrade, researchers can zoom in on matter with unprecedented depth — down to less than a nanometer, or more than 50,000 times smaller than the width of a human hair. They can also capture processes at ultrafast speeds, like taking a rapid-fire snapshot of nature in motion in less than one-billionth of a second.
Meanwhile, across the Argonne campus, the ALCF is installing Aurora. A behemoth of a supercomputer, it will not only interpret the details of all of those images and return them to researchers with great efficiency, but also help steer experiments at Argonne.
One of the first exascale computers in the world, Aurora will be capable of performing two billion billion calculations per second, making it nearly 50 times faster than ALCF’s current supercomputer, Polaris.
Aurora comprises refrigerator-sized cabinets that house more than 10,000 nodes, the miniature workhorses of the machine. Each node contains a combination of processing units to help Aurora perform many functions simultaneously. And these processors, combined with powerful networking and high performance storage, allow all those nodes to work together for greater speed and efficiency.
One of Aurora’s distinguishing features will be its ability to seamlessly integrate the important scientific tools of data analysis, modeling and simulation, and AI. This trifecta produces a feedback mechanism that plays a significant role in redirecting the design of new simulations and experiments.
Introducing such immense computational power will help researchers construct infinitely more accurate models in a diversity of scientific domains and complex dynamic systems, such as climate, battery storage and cosmic evolution.
But even with the exascale computing power of Aurora, the data rates produced by scientific instruments are becoming so large that conventional approaches to analyzing data will still take too long.
Researchers are looking toward novel ways to reduce and process data in intervals close to real time. This will combine new methods in AI with traditional approaches such as modeling and simulation, image processing and statistical analysis.
One key AI technique is called machine learning, which creates models that get more accurate as they acquire more data — no problem there: The mountain of data from APS will be used to train AI models on Aurora.
With the ability to highlight important features in the data, these AI models can significantly speed up computations of complex systems by as much as 1,000 times and, eventually, be used to make predictions. They can also expedite some of the most intensive computational tasks, boosting scientists’ ability to unravel complex phenomena, like those that occur when a battery charges or when solar panels convert sunlight into electricity.
“This is critical if we want to steer or autonomously guide an experiment,” said Nicholas Schwarz, Scientific Software Engineering and Data Management group leader at the APS, who is responsible for the collaboration between APS and Aurora. “The experiments are so complex, the data so big, events happen so quickly, that humans can’t interpret them fast enough. This combination of Aurora, the upgraded APS and AI will present opportunities in scientific exploration at speeds and scales previously inaccessible to science.”
Transformative science
Jonathan Almer’s team at the APS is among the many groups that will benefit from this near-real-time approach provided by the collaboration. The team supports industry and military collaborators studying various alloys for use in aerospace and nuclear applications. These scientists want to see how their materials react to external stimuli, such as elevated temperatures and mechanical loads.
Carefully simulating those conditions at the beamline, high-energy X-ray beams explore the structure of the material and create a large cache of 2D images, often producing mountains of data.
“Researchers want us to reconstruct this extremely fast so they can have an image of their sample to help them decide which load or temperature to go to next,” explained Almer, group leader in Argonne’s X-ray Science division.
To handle the data explosion, the X-ray detectors that capture these images will stream this data to Aurora, which delivers highly detailed 3D reconstructions of the sample. Then it will filter results back to APS experimenters.
This rapid turnaround will allow researchers to better home in on pivotal pieces of information, quickly narrow the focus of their experiments, and accelerate discovery.
The combined power of Aurora and APS will also allow scientists to map brain structure and function at an unprecedented scale, probing large brain volumes in a single experiment. Like deciphering a map, scientists are already plotting the axons and neurons, or highways and arteries, of a brain’s communication pathways.
Work has focused primarily on the smaller mouse brain, but ultimately researchers would like to zoom in on the back streets and alleys of the larger, more complicated human brain.
But building a complete road map of the entire human brain will produce a deluge of data, much of it packed into the innumerable X-ray images produced at the upgraded APS. Ciphering through the data to ferret out the billions of connections will require the integration of Aurora and AI algorithms to make sense of it all.
“Let’s say we did a nano X-ray of an entire primate,” said Narayanan “Bobby” Kasthuri, a neuroscience researcher at Argonne and the University of Chicago. “Essentially, we would have so many axons, there’s not enough time or humans to trace every one of them manually. So, we work with our collaborators at the ALCF to come up with AI algorithms that will automatically trace these axons throughout the brain.”
By studying the structure and connectivity between cells within the brain, scientists aim to understand the relationship between neural structure and mental function.
“All told, this confluence of technologies, the massive increase in computing from Aurora and AI and the brighter, higher energy, more coherent X-rays of the APS, will enable a transformation in the way we do science,” Schwarz noted. “On their own, we could not successfully conduct science on these grand scales. But when we combine them, a whole new world of science is open to us.”
Aurora is currently being installed at Argonne and will start up in 2023. The upgraded APS is scheduled to come online in 2024.
The Argonne Leadership Computing Facility provides supercomputing capabilities to the scientific and engineering community to advance fundamental discovery and understanding in a broad range of disciplines. Supported by the U.S. Department of Energy’s (DOE’s) Office of Science, Advanced Scientific Computing Research (ASCR) program, the ALCF is one of two DOE Leadership Computing Facilities in the nation dedicated to open science.
About the Advanced Photon Source
The U. S. Department of Energy Office of Science’s Advanced Photon Source (APS) at Argonne National Laboratory is one of the world’s most productive X-ray light source facilities. The APS provides high-brightness X-ray beams to a diverse community of researchers in materials science, chemistry, condensed matter physics, the life and environmental sciences, and applied research. These X-rays are ideally suited for explorations of materials and biological structures; elemental distribution; chemical, magnetic, electronic states; and a wide range of technologically important engineering systems from batteries to fuel injector sprays, all of which are the foundations of our nation’s economic, technological, and physical well-being. Each year, more than 5,000 researchers use the APS to produce over 2,000 publications detailing impactful discoveries, and solve more vital biological protein structures than users of any other X-ray light source research facility. APS scientists and engineers innovate technology that is at the heart of advancing accelerator and light-source operations. This includes the insertion devices that produce extreme-brightness X-rays prized by researchers, lenses that focus the X-rays down to a few nanometers, instrumentation that maximizes the way the X-rays interact with samples being studied, and software that gathers and manages the massive quantity of data resulting from discovery research at the APS.
This research used resources of the Advanced Photon Source, a U.S. DOE Office of Science User Facility operated for the DOE Office of Science by Argonne National Laboratory under Contract No. DE-AC02-06CH11357.
Argonne National Laboratory seeks solutions to pressing national problems in science and technology. The nation’s first national laboratory, Argonne conducts leading-edge basic and applied scientific research in virtually every scientific discipline. Argonne researchers work closely with researchers from hundreds of companies, universities, and federal, state and municipal agencies to help them solve their specific problems, advance America’s scientific leadership and prepare the nation for a better future. With employees from more than 60 nations, Argonne is managed by UChicago Argonne, LLC for the U.S. Department of Energy’s Office of Science.
The U.S. Department of Energy’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science.
