Preserving historic Wisconsin aerial photos with a little help from CHTC

Associate State Cartographer Jim Lacy works with CHTC to digitize and preserve historical aerial photography for the public.

Right now, hundreds of thousands of historic aerial photos from around Wisconsin are gradually aging in file cabinets on the University of Wisconsin-Madison campus, with some of the photos approaching 100 years old. Although historical photography is a captivating and well-trodden method to study the past, without intervention, this opportunity will be lost as the photos get older and begin to decay.

Addressing this challenge is the State Cartographer’s Office (SCO) and the Arthur H. Robinson Map Library (RML), units within the Department of Geography at the University of Wisconsin-Madison, who are working to digitally preserve Wisconsin aerial photography from the twentieth century. The SCO and RML team created a free digital tool in 2011 called the Wisconsin Historic Aerial Image Finder (WHAIFinder), where the public can view and download digital versions of the air-photos at any time. The platform currently provides almost 40,000 Wisconsin aerial images, ranging from 1937-1941.

SCO’s Associate State Cartographer Jim Lacy continues the effort of digitizing Wisconsin air-photos from other decades alongside Map & Geospatial Data Librarian Jaime Martindale from the RML. “We really want to work hard to digitally preserve all of that photography. That way it’s available forever,” Lacy said.

One of the steps necessary when digitizing the photography is to convert the images to Cloud Optimized GeoTIFF (COG) format and generate jpegs. This caused a computing bottleneck for Lacy, who experimented with his local PC and found that about 100,000 images in need of converting would take over a month to process. “What we’re doing with the COG conversion frankly is not that complicated.” Lacy said, “It’s basically reformatting data, but it’s still fairly compute intensive.”

Asking himself if there was a better way, Lacy went in search of a solution and looked to the Center for High Throughput Computing (CHTC) last November for a more efficient computing option. Specializing in high throughput computing (HTC), CHTC allows for users like Lacy to split their work up into a large amount of smaller-scale jobs that can be processed in parallel. He attended a CHTC workshop and worked in close collaboration with the CHTC facilitation team to find the right computing techniques.

“The facilitators were extremely helpful in giving me pushes in the right direction,” Lacy remarked. He found that using the HTCondor Software Suite (HTCSS) was a “learning curve,” despite his previous experience with necessary user elements like shell scripting and UNIX. “It took some learning, patience, and a lot of trial and error.”

The impact of using CHTC services and capacity was noteworthy. Running his own case study using input files from the National Agriculture Imagery Program (NAIP) for Dane County, Lacy found that what took his local PC 93 minutes to run took five minutes when done through CHTC. “It’s been a huge time saver,” Lacy stated. He also found that utilizing CHTC allowed room for mistakes and experimentation. “If we were to use a desktop PC it would take a week each time. For us, the option of repeatability is really important.”

One issue Lacy had while using CHTC pertained to the data transfer time, despite the entire process taking less time than his local PC. In his case study, the total data transfer overhead came out to around 21 hours. That was 14 times longer than the amount of time it took to process the data. Lacy recently met with CHTC, including members of the facilitation team and the Pelican project to discuss possible improvements to data transferring, as well as making the entire process less hands-on.

Utilizing open capacity from a research computing center, Lacy views his work with the SCO to be atypical to the research world. “We do some research, but our main focus is outreach and connecting people to mapping related resources. We’re all about the Wisconsin Idea,” Lacy remarked. “The goal of the Wisconsin Idea is to share our knowledge and help other folks solve problems. ”