Neural networks learn faster using ETH software

Two researchers from the Scalable Parallel Computing Lab at the Swiss Federal Institute of Technology in Zurich (ETH) have developed a software solution to rapidly speed up the training of deep learning applications. This is important as this process is the most resource-​demanding and costly step of all, ETH Zurich writes in a press release. It accounts for up to 85 percent of the training time. For example, a single training run of a sophisticated voice recognition model can cost around 10 million US dollars.

The new software named NoPFS was developed by Roman Böhringer and Nikoli Dryden. It uses the concept of clairvoyance: “It exploits a pseudo-​random and therefore predictable process in training more effectively than other tools have to date”, ETH Zurich writes in the press release. NoPFS is able to predict the sequencing of data samples. Nikoli Dryden explains that this knowledge means that “the individual data items can be loaded in advance based on the frequency they will be accessed”, which allows the data items to be “prefetched to the respective processors, which speeds up training”.

Depending on the respective supercomputer, the number of graphics processors used and the size of the data set, NoPFS was able to speed up the training of neural networks by a factor of between 2.1 and 5.4. With AI-capable systems, such as those used in the new Alps supercomputer, the training efficiency increase may be somewhat lower. In spring 2023, Alps is set to replace the previous flagship among Swiss supercomputers, Piz Daint, at CSCS, the Swiss National Supercomputing Center that is connected to ETH Zurich.