Fermilab’s NOvA experiment relies on the massive computing power of the Open Science Grid (OSG) in its quest to see how neutrinos change over a 500 mile journey
“High Energy Physics experiments are trying to understand the basic building blocks that make up our universe,” says Alex Himmel. Himmel is the newly installed computing coordinator for the ground-breaking NOvA experiment at the Fermi National Accelerator Laboratory (Fermilab), near Batavia, Illinois. NOvA (NuMI Off-Axis νe Appearance) is designed to detect neutrinos in Fermilab’s NuMI (Neutrinos at the Main Injector) beam. Recently, the NOvA researchers saw their first big breakthrough when the detector revealed the first evidence of oscillating neutrinos fired from 500 miles away.
Photo Courtesy Alex Himmel
Illustration: Fermilab; used with permission. Above is a graphic representation of one of the first neutrino interactions captured at the NOvA far detector in northern Minnesota. The dotted red line represents the neutrino beam, generated at Fermilab in Illinois and sent through 500 miles of earth to the far detector. The image on the left is a simplified 3-D view of the detector – the top right view shows the interaction from the top of the detector, and the bottom right view shows the interaction from the side of the detector.
While neutrinos are the most abundant known particles in the universe, their detailed properties are poorly understood. They come in three forms: muon neutrinos, electron neutrinos, and tau neutrinos. NOvA seeks to help discover what causes neutrinos to transform—oscillate—from one form to another. Understanding what causes the oscillation of a muon neutrino into an electron neutrino could help us understand those basic building blocks (in other words, elementary particles) of the universe.
“If we want to study neutrinos carefully, we have to build these specialized experiments like NOvA to probe directly into the weak nuclear force,” says Himmel. “Why do we care about the weak nuclear force? We need to understand why we are in a universe that is all matter instead of antimatter. We believe it has to do with the weak nuclear force.”
The $278-million NOvA project is an international collaboration of nearly 210 scientists and engineers from 39 universities, laboratories, and other institutions. The 14,000-ton, multi-story NOvA detector is located in a remote area near Ash River, Minnesota. Neutrinos are generated at Fermilab in Illinois and beamed to the detector over 500 miles away. The beam first passes through an underground near detector, which measures the beam’s neutrino composition before it leaves the Fermilab site. Neutrinos pass through matter as though it doesn’t exist.
“Not that many muon neutrinos become electron neutrinos,” says Himmel. “We are looking for a really small sample. Plus, neutrinos interact very rarely. In this first result, we saw only six. We must use computationally intensive techniques to identify these interactions to tell if they really are electron neutrinos and not something else from the background.”
“This is where the Open Science Grid plays such an important role for us,” says Himmel. “We would not have been able to do the first analysis without the OSG. It allows us to look at unbiased samples earlier and understand them earlier. Since this is a new detector, there are many things to understand. OSG allows for detailed simulations of the detector and thus helps with understanding the behavior of the detector.”
Most neutrino detectors are underground to shield them from cosmic rays. For NOvA, Himmel says the electronics became advanced enough that it could be built above ground. This saves a lot of money, but it means the detector is being hit with cosmic rays all the time. With only a handful of neutrinos appearing, large computing power is needed to filter out the cosmic rays to find the rare neutrino signals. By reprocessing the data quickly, they reduce wait time by months and can fine-tune analysis.
“The cosmic ray rate at the far detector is 100 kHz, or 100,000 every second,” says Himmel. “That’s almost 8 billion every day. Even though we use the times from the pulsed beam at Fermilab so we know where to look, we typically have 50 cosmic rays close in time (<0.55 ms) to every neutrino we see. Hence the need for serious computing power to search through them for the neutrinos!”
“The OSG infrastructure not only takes advantage of opportunistic computing resources,” adds Himmel, “but it also provides the infrastructure used at Fermilab itself, and enables our partner universities (e.g. FZU) to contribute their resources. Using OSG tools and services allows us to transparently compute across resources at FNAL, our partners, and opportunistic resources elsewhere.”
NOvA also coordinates with other neutrino experiments. Together, they hope to better understand whether neutrinos behave like other particles in the Standard Model.
“The headline goal for us is to study the nature of neutrino oscillations (how matter causes neutrinos to change forms),” says Himmel. “The neutrinos are interacting with matter as they travel. We’re looking at the differences of how neutrinos and anti-neutrinos travel through the earth, and trying to extract the mass hierarchy. Two neutrinos are close in mass and the third is very different. We don’t know if the third is heavier or lighter – we just know its mass is very different.”
Alex Himmel had his first job at Fermilab as a summer student at the age of 16. He’s been investigating particle physics ever since. He started focusing on neutrinos in grad school, previously worked on the MINOS experiment as a doctoral student, and recently returned to Fermilab as a Wilson Fellow.
For this breakthrough experiment, NOvA used 10 million CPU hours on the OSG from April 28 to July 31:
Graph provided by Bo Jayatilaka, Fermilab
Graph provided by Bo Jayatilaka, Fermilab
-Article by Greg Moore