About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Publication
IEEE SPM
Paper
Distributed Training of Deep Neural Network Acoustic Models for Automatic Speech Recognition: A comparison of current training strategies
Abstract
The past decade has witnessed great progress in automatic speech recognition (ASR) due to advances in deep learning. The improvements in performance can be attributed to both improved models and large-scale training data. The key to training such models is the employment of efficient distributed learning techniques. In this article, we provide an overview of distributed training techniques for deep neural network (DNN) acoustic models used for ASR. Starting with the fundamentals of data parallel stochastic gradient descent (SGD) and ASR acoustic modeling, we investigate various distributed training strategies and their realizations in high-performance computing (HPC) environments with an emphasis on striking a balance between communication and computation. Experiments are carried out on a popular public benchmark to study the convergence, speedup, and recognition performance of the investigated strategies.