Adventures at IceCube

Computing at the South Pole is always interesting and can be quite challenging. The IceCube detector, when complete, will be made up of 80 strings of digital optical modules (DOMs) drilled deep into the polar ice sheet. Each string contains 60 optical modules spaced evenly from a depth of 1.5 km to 2.5 km.  An additional six strings have been proposed with closer spacing to provide better sensitivity to low energy particles. An array of DOMs at the surface, called IceTop, provides insight into atmospheric events.

Each of the DOMs connects to communications cables that terminate in the IceCube Lab, a dedicated building located in the center of the detector. Each string connects to a dedicated computer called a DOM Hub. A DOM Hub is a 4U chassis housing a single-board computer and eight custom PCI cardsthat  communicate with the DOMs. Data acquisition software (the DAQ) reads data from the DOM Hubs and arranges it into events. This is the raw data stream from the detector.

The raw data stream is currently about 16MB/s, or about .5 PB/year. Connectivity to the pole happens via satellites and is expensive, which constrains the bandwidth available for data transfer. IceCube is allotted about 70GB/day for data transfer to the IceCube data warehouse at Madison, Wisconsin. That’s about an order of magnitude less than the raw data volume produced by the detector, so a group of twenty-five machines grind through the data and select the “most interesting” events to be transferred over the satellite. A copy of that data is written to tape for safety purposes. In addition, the raw data stream is written to tape. These tapes are sent north every year when the station opens in November. In principal, the raw data should not be needed again, as it is mostly background noise, but bugs are found and the data is usually re-examined.

So all told, we maintain about 140 computers in the IceCube lab. Power consumption is currently between 30 and 40 kilowatts and is projected to peak between 50 and 60 kilowatts for the full detector. The computers are housed in a single room large enough for 24 standard racks. Ironically, one of our biggest problems is cooling. The answer is obvious: open the window; that’s essentially what we do. Motorized louvers regulate the flow of outside air into the building. This air is mixed with recirculated warm air and ducted into the lab. Everything is fine as long as nothing breaks, and for the most part, it doesn’t; however, we have had two incidents in the last three years in which the louvers closed completely and the temperatures rose to over 110F (44C).

Power is provided by the diesel generators at the Amundson-Scott station. This generally works pretty well, but it’s best to be prepared for outages. Each rack has a UPS that is capable of keeping the machines running for at least 20 minutes. Typically, outages or brown outs are short, but they do happen on a fairly routine basis.

The other killers in the environment are altitude and humidity. The elevation is 9300 feet. The cold air ensures that humidity is basically zero. The thin, dry air exacerbates the cooling issues and increases the ESD risks dramatically.

As may be obvious at this point, all of the equipment is operated either out of, or just within specifications. Despite that, it has performed well. Hard drive failures are a bit more frequent than normal, and you do have to be careful about grounding yourself appropriately, but otherwise, things seem to behave as expected.

Perhaps the hardest parts are logistical. The station is only accessible during the austral summer, which ranges from early November to late February. From March to October no cargo moves in or out and you have to live with what you have. Space is limited, so the natural desire to spare aggressively has to be tempered to fit within the available space. Cargo comes in on planes, and if we’re lucky, it is transported from the cargo area to the IceCube Lab (a distance of about 1 km) by heavy equipment. Even so, it must then be unloaded by hand and ultimately carried up a flight of stairs to the machine room. UPSes are among the most dreaded items you can receive.

You also begin to appreciate how many problems overnight shipping solves. Even when you plan as carefully as you can, there are always some little things that get missed. Many times these things can be hand-carried by people traveling to the pole. It’s usually a few weeks at best before these items arrive. Bigger items require lots of logistical magic and might require some creative scheduling to ensure all of your goals are met by station close.

And finally, the winter comes. The station closes and the sun sets for six months. During the winter, IceCube keeps two people on station (Winter Overs). These hardy souls keep the computers and the detector running throughout the winter. This means at least one walk daily from the station to the IceCube lab (1 km distance), so long as it is safe to do so. Temperatures drop to -90F or lower, and it is dark and windy. I can only imagine what that walk must be like.

In May of 2009 I was scheduled to deploy to the South Pole to participate in the maintenance and ongoing construction of the IceCube neutrino detector. In my case, this takes me away from the aspects of the project with which I am most familiar (cluster and grid computing), and instead gets me working more closely with the detector and its supporting computing systems. The process of deploying to the South Pole begins about six months before your actual deployment. The geographic South Pole is at an elevation of 9300ft. Varying temperatures any atmospheric pressure create an effective altitude of over 10,000ft. Aircraft is the primary method of access (with the exception of some hardy souls who make the overland trek). The Amundson-Scott station has a medical facility and at least one doctor on station, but the nearest full medical facilities are at least 12 hours away. In order to limit medical problems on station, all participants undergo a fairly extensive medical and dental examination to ensure that they are physically qualified (abbreviated PQ).

The primary health risks are altitude sickness, extreme cold (-40F summer temperatures), and snow blindness. Altitude sickness is the primary concern as there is no real way to predict who will be affected unless you’ve had it before (in which case, you will most likely get it again). It is fatal without medical treatment and evacuation to a lower altitude. The PQ process screens for conditions which could be confused with, or cause complications to altitude sickness. You are also checked for conditions which might develop into emergency situations while you are deployed.

If you get a clean bill of health (you are PQ) your tickets are booked and you are ready to go. As a participant in the US Antarctic Program (USAP), I travel through Christchurch, New Zealand to reach the pole. The USAP operates a center near the Christchurch airport which has an Antarctic museum and also the Clothing Distribution Center (CDC). The CDC is where you pick up your “Extreme Cold Weather” gear (ECW) and get any final items crossed off the list (for example, a flu shot if you did not get it during the PQ physical). You are issued all of the gear you need to operate during the South Pole austral summer when temperatures typically vary between -50F and -10F. You try everything on to make sure it fits well and is in good order, and then pack it up in preparation for your trip.

This is where things can get interesting. The trip to the pole typically happens in two stages: Christchurch to McMurdo station, and McMurdo to the pole. The first week of November is still fairly early in the summer season, so the weather fluctuates wildly and flight delays are common. In some cases, a flight will take off only to turn around because the weather will not permit a safe landing. This is called a boomerang, and it is no fun as you will typically end up on a flight the following day. That’s a lot of pretty tedious travel. On the day of departure, you go back to the CDC and get ready for your flight. The flights are operated by the military (in our case, US Air Force), but for us follow a similar pattern to boarding a normal flight. After security screenings and and baggage check (they weigh both you and your luggage) you wait to board the plane. They want you to wear, or at least carry your ECW gear onto the plane, so you can end up a bit over dressed while waiting for the plane.

I got lucky and got through without any boomerangs or delayed flights. The first leg from Christchurch to McMurdo was a C-17 cargo plane. From McMurdo to the pole, we traveled in a HERC LC-130 cargo plane. It was interesting to see the inside of these planes as they have no interior walls so all of the mechanical and electrical systems are exposed. They are also very loud and we are issued ear plugs to preserve our hearing and our sanity. Upon arrival at McMurdo we are shuttled to the mess hall for a briefing covering some of the basics of life at the station. Conservation and recycling are a big deal as is general safety and situational awareness (stay on the marked trails, pay attention to the weather). For those of us deploying to the pole we are given a bit of extra medical advice about hazards at the pole, and are issued a drug to help acclimate to the higher altitude.

The time at McMurdo was really pretty busy for us. We were given our room assignments and issued bedding. A quick bite to eat and off to “bag drag” which is the preparation for the flight the next day (you and all of your luggage are weighed again). We were lucky as our flight was scheduled to leave the following day. Most of our gear was left in the building from which we would depart so there was no need to drag it all over the base. The temperatures were fairly mild and it was reasonable to walk around in nothing more than a fleece jacket and light shoes at least for short trips.

The next day we were shuttled to our plane, which miraculously took off on schedule …

IceCube is a telescope being constructed at the South Pole that will search for neutrinos–chargeless and elusive particles that results from some types of radioactive decay or nuclear reactions.  Neutrinos may form, for example, when stars explode or when gamma-rays burst, and when other things happen that involve infamous objects like black holes and neutron stars. [Read more about IceCube].

Steve Barnet, an HPC Administrator at the University of Wisconsin, traveled down to the South Pole on November 6, 2009. Steve is a who plans to use the Open Science Grid to run computer simulations and analysis for IceCube.