A Mechanism for Distributed Deep Learning Communication Optimization

Intensive communication and synchronization cost for gradients and parameters is the well-known bottleneck of distributed deep learning training. Based on the observations that Synchronous SGD (SSGD) obtains good convergence accuracy while asynchronous SGD (ASGD) delivers a faster raw training speed, we propose Several Steps Delay SGD (DeSGD) to combine their merits, aiming at tackling the communication bottleneck via communication sparsification. DeSGD explores both global synchronous updates in the parameter servers and asynchronous local updates in the workers in each periodic iteration. The periodic and flexible synchronization makes DeSGD achieve good convergence accuracy and fast training speed. To the best of our knowledge, we strike the new balance between synchronization quality and communication sparsification, and improve the trade-off between accuracy and training speed. Specifically, the core components of DeSGD include proper warm-up stage, steps delay stage, and our novel algorithm of global gradient for local update (GGLU). GGLU is critical for local update operations to effectively compensate the delayed local weights. Furthermore, we implement DeSGD on MXNet framework and comprehensively evaluate its performance with CIFAR-10 and ImageNet datasets. Experimental results show that DeSGD can accelerate distributed training speed under different experimental configurations, by up to 110%, while achieving good convergence accuracy.