We consider distributed optimization under communication constraints for training deep learning models. We propose a new algorithm, whose parameter updates rely on two forces: a regular gradient step, and a corrective direction dictated by the currently best-performing worker (leader). Our method differs from the parameter-averaging scheme EASGD in a number of ways: (i) our objective formulation does not change the location of stationary points compared to the original optimization problem; (ii) we avoid convergence decelerations caused by pulling local workers descending to different local minima to each other (i.e. to the average of their parameters); (iii) our update by design breaks the curse of symmetry (the phenomenon of being trapped in poorly generalizing sub-optimal solutions in symmetric non-convex landscapes); and (iv) our approach is more communication efficient since it broadcasts only parameters of the leader rather than all workers. We provide theoretical analysis of the batch version of the proposed algorithm, which we call Leader Gradient Descent (LGD), and its stochastic variant (LSGD). Finally, we implement an asynchronous version of our algorithm and extend it to the multi-leader setting, where we form groups of workers, each represented by its own local leader (the best performer in a group), and update each worker with a corrective direction comprised of two attractive forces: one to the local, and one to the global leader (the best performer among all workers). The multi-leader setting is well-aligned with current hardware architecture, where local workers forming a group lie within a single computational node and different groups correspond to different nodes. For training convolutional neural networks, we empirically demonstrate that our approach compares favorably to state-of-the-art baselines. This work is a gentle extension of [2].
翻译:在培训深层学习模式的沟通限制下,我们考虑分配优化。我们提出一个新的算法,其参数更新依赖于两种力量:一个定期梯度步骤,以及由目前表现最佳的工人(领导人)决定的纠正方向。我们的方法在许多方面不同于参数稳定计划ESGD(ESGD ) :(一) 我们的客观提法不会改变固定点的位置,而不会改变最初优化问题;(二) 我们避免通过拉动当地工人降至不同地方最低水平(即其参数的平均值)而导致的趋同减速;(三) 通过设计,我们更新打破了对称的诅咒(目前表现为低度、低度、低度、低度、低度、低度。 (四) 我们的方法提高了沟通效率,因为它只播放了领导者而不是所有工人的参数;(二) 我们对拟议算法的批次进行理论分析,我们称之为 " 领先者梯度 " (LGD), 其平均参数的变量(LSGDDD) ;(三) 最后,我们用设计方式打破了对平衡的诅咒的诅咒, 亚精度, 亚精度的亚化的亚化的亚化的亚化的亚化, 将我们每个工人的每个工人的每一组展示一个代表着一个方向, 向一个方向, 向一个代表着一个最有色的工人的系统, 其最有色的系统。