Due to limited communication resources at the client and a massive number of model parameters, large-scale distributed learning tasks suffer from communication bottleneck. Gradient compression is an effective method to reduce communication load by transmitting compressed gradients. Motivated by the fact that in the scenario of stochastic gradients descent, gradients between adjacent rounds may have a high correlation since they wish to learn the same model, this paper proposes a practical gradient compression scheme for federated learning, which uses historical gradients to compress gradients and is based on Wyner-Ziv coding but without any probabilistic assumption. We also implement our gradient quantization method on the real dataset, and the performance of our method is better than the previous schemes.
翻译:由于客户的通信资源有限和大量模型参数,大规模分布式学习任务受到通信瓶颈的影响。渐进压缩是通过传输压缩梯度来减少通信负荷的有效方法。在随机梯度下降的情况下,相邻各轮之间的梯度可能具有高度的相关性,因为他们希望学习同样的模型,本文提出了联邦学习的实用梯度压缩计划,该计划使用历史梯度压缩梯度,以Wyner-Ziv编码为基础,但没有任何概率性假设。我们还在真实数据集上采用了梯度定分法,我们方法的性能优于以往方法。