For a gargantuan nuclear physics experiment that will generate big data at unprecedented rates—called A Large Ion Collider Experiment, or ALICE—the University of Tennessee has worked with the Department of Energy’s Oak Ridge National Laboratory to lead a group of U.S. nuclear physicists from a suite of institutions in the design, development, mass production and delivery of a significant upgrade of novel particle detectors and state-of-the art electronics, with parts built all over the world and now undergoing installation at CERN’s Large Hadron Collider (LHC).
“This upgrade brings entirely new capabilities to the ALICE experiment,” said Thomas M. Cormier, project director of the ALICE Barrel Tracking Upgrade (BTU), which includes an electronics overhaul that is among the biggest ever undertaken by DOE’s Office of Nuclear Physics.
ALICE’s 1,917 participants from 177 institutes and 40 nations are united in trying to better understand the nature of matter at extreme temperature and density. To that end, the LHC creates a succession of “little bangs”—samples of matter at energy densities not seen in the universe since microseconds after the Big Bang. ALICE’s detectors identify the high-energy particles and track their trajectories, interactions and decays that produce lower-energy daughter particles, daughters of daughters, and so on. The upgrades enable ALICE to more efficiently track particles at high rates, digitize their weak analog electronic signals continuously and stream the tsunami of readout data to high-performance computing (HPC) centers around the world for analysis.
“Revising the instrumentation lets us expand the window of the science that ALICE can look at,” said Cormier, who is a physicist at ORNL and professor at the University of Tennessee at Knoxville. “A lot of things are waiting out there to be discovered if we just have the sensitivity to see them.” Combined with upgrades to the LHC accelerator, the BTU will increase sensitivity tenfold, enabling greater differentiation of the underlying science.
Completed ahead of schedule and under budget, the project relied on participants from DOE’s Oak Ridge (ORNL) and Lawrence Berkeley (LBNL) National Laboratories and seven universities: California at Berkeley, Creighton, Houston, Tennessee at Knoxville (UTK), Texas at Austin (UT Austin), Wayne State and Yale.
The upgrade effort began in April 2015 and ended in November 2019, delivering a suite of advanced detectors and electronics to CERN. Researchers anticipate the completion of installations this spring.
Considering the scale, this is no easy feat. Sited underground at the Franco-Swiss border, ALICE is heavier than the Eiffel Tower. A 52-foot-tall magnet is its front door. Behind it, nuclear physicists have rolled out one of the world’s biggest barrel instruments, housing many detectors arranged in concentric cylinders LHC’s beam line runs through its center axis.
Significant effort went into improving two ALICE detector systems. One is the Time Projection Chamber (TPC), a gas-filled cylindrical apparatus the size of a shuttle bus. As charged particles speed through the gas, a magnetic field bends their paths, creating curved trajectories that reveal their momenta and masses and, in turn, their identities. Each endcap of the TPC cylinder is covered with two concentric rings of novel inner and outer readout chambers that receive the ionization charge and amplify it using an innovative four-layer system of micro-pattern perforated Gaseous Electron Multiplier foils. A system of nearly a half million, millimeter-scale pads spreads across the ends of the TPC cylinder to collect the amplified charge and create an electronic image of the charged particle tracks.
The second detector system to receive an upgrade is a seven-layer Inner Tracking System. LBNL collaborated with UT Austin to develop its middle layers, which include a strong-but-lightweight carbon-fiber frame to support seven layers of staves holding 24,000 silicon-pixel sensors for high-precision particle tracking. Each pixel is 30 × 30 micrometers squared—finer than an average human hair. This detector will have a total of 12.5 billion pixels—making it the largest “digital camera” ever built.
Processing the biggest of data
The upgrade dramatically increased the number of events per second that ALICE can sample and read out. Kenneth Read, manager of BTU’s electronics upgrade, led a huge undertaking in design, fabrication and assembly of electronics hardware. Read, an experimental nuclear physicist with expertise in high performance computing, holds joint appointments at ORNL and UTK.
Ultimately, Read’s team delivered 3,276 circuit boards (plus 426 spares) for readout of the half a million TPC channels. The electronics upgrade makes it possible to digitize and distribute 5 million samples per second per channel.
“Non-stop data output totaling 3 terabytes per second will flow from the Time Projection Chamber, 24/7, during data taking,” Read explained. “Historically, many experiments have dealt with megabyte per second, or even gigabyte per second, data rates. Real-time processing of streaming scientific data at 3 terabytes per second is approaching unique in the world. This is a big data problem of immense proportions.”
That data provides a snapshot of the quantum system known as the quark–gluon plasma—the matter of the very early universe first discovered at the Relativistic Heavy Ion Collider (RHIC) at Brookhaven National Laboratory and subsequently studied at both RHIC and the ALICE detector at the LHC. Such a plasma is produced here on Earth when a powerful collider, such as the LHC, accelerates heavy ions, each containing many protons and neutrons, and collides these heavy ions with so much energy that their protons and neutrons “melt” into their elementary building blocks—quarks and gluons—in a plasma more than 100,000 times hotter than our sun’s core. This exploding “soup” of liberated quarks and gluons forms particles that decay into myriad other particles. The detector array identifies and maps them so nuclear scientists can reconstruct what happened and gain understanding of the collective phenomena.
Capturing that plethora of particle collision events required a team of institutes to develop a custom-tailored chip that could digitize and read out the biggest of data. Enter “SAMPA.” At the heart of ALICE’s massive electronics upgrade, this chip began as the PhD thesis project of Hugo Hernandez, then at the University of Sao Paolo.
SAMPA chips and other electronic components were shipped to Zollner Electronics in Silicon Valley for assembly onto printed circuit boards fabricated by electronics manufacturing giant TTM Technologies. The team of ORNL PhD-level electrical engineers making critical contributions throughout the electronics upgrade—lead designer Charles Britton with N. Dianne Bull Ezell, Lloyd Clonts, Bruce Warmack and Daniel Simpson—also developed a high-throughput station to test the boards right at the assembly factory. Whereas it traditionally took 1 hour to diagnose and debug a complex board, the ORNL team’s automated process did it in a mere 6 minutes.
“It used to be, you’d order a thousand widgets, receive them at Oak Ridge and test them,” Read reminisced. “You’d send the bad ones back to the factory and the good ones on to CERN.” The ORNL test stations allowed the assembly factory to ship passing boards directly to CERN in small “just-in-time” batches for quicker installation than possible when waiting on large lots.
The researchers will calibrate the BTU using cosmic rays. Then, the upgraded equipment will be ready for the high-luminosity LHC Run-3, anticipated in 2021. Several runs of various collision data sets are planned—lead-on-lead, proton-on-lead and proton-on-proton—to illuminate emergent features of the quark-gluon plasma.
Even one year of collected raw data will be far too big to archive. The readout system winnows the streaming data to petabyte scale by processing it on the fly with hardware acceleration using field-programmable gate arrays and graphics processing units (GPUs)—considered a best practice. The reduced data is distributed over high-speed networks to HPC centers around the world, including ORNL’s Compute and Data Environment for Science, for further processing. As experiments get larger, physicists build the case for also using centralized resources, such as the Oak Ridge Leadership Computing Facility’s Summit supercomputer for GPU-accelerated data processing.
“Other large experiments at the LHC using different particle detectors—notably ATLAS and CMS—will confront some of the same data challenges as ALICE in 2027 and beyond,” said ALICE researcher Constantin Loizides of ORNL. “The world-leading capabilities of the BTU electronics will likely benefit future physics experiments like the planned electron–ion collider, a top priority for U.S. nuclear physics.”
The Department of Energy’s Office of Science funded the U.S. contribution to the BTU.
UT-Battelle manages ORNL for DOE’s Office of Science. The single largest supporter of basic research in the physical sciences in the United States, the Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit https://science.energy.gov/. —by Dawn Levy
Original post https://alertarticles.info
