Communication efficiency is crucial in federated learning. Conducting many local training steps in clients to reduce the communication frequency between clients and the server is a common method to address this issue. However, the client drift problem arises as the non-i.i.d. data distributions in different clients can severely deteriorate the performance of federated learning. In this work, we propose a new SGD variant named as DOMO to improve the model performance in federated learning, where double momentum buffers are maintained. One momentum buffer tracks the server update direction, while the other tracks the local update direction. We introduce a novel server momentum fusion technique to coordinate the server and local momentum SGD. We also provide the first theoretical analysis involving both the server and local momentum SGD. Extensive experimental results show a better model performance of DOMO than FedAvg and existing momentum SGD variants in federated learning tasks.
翻译:通信效率在联谊学习中至关重要。 在客户中采取许多当地培训步骤以减少客户与服务器之间的通信频率,是解决这一问题的共同方法。然而,客户的漂移问题产生的原因是,不同客户中的数据分布会严重恶化联谊学习的绩效。在这项工作中,我们提议一个新的SGD变式,称为DOM,以提高联谊学习的模型性能,在这种学习中保持双动缓冲。一个势头缓冲跟踪服务器的更新方向,而另一个则跟踪本地更新方向。我们采用了一种新的服务器动力聚合技术,以协调服务器和本地动力 SGD。我们还提供了第一次涉及服务器和本地动力SGD的理论分析。广泛的实验结果显示DOM比FDAvg更好的模型性性能,在联谊学习任务中现有的动力 SGD变式。