Federated learning is a machine learning training paradigm that enables clients to jointly train models without sharing their own localized data. However, the implementation of federated learning in practice still faces numerous challenges, such as the large communication overhead due to the repetitive server-client synchronization and the lack of adaptivity by SGD-based model updates. Despite that various methods have been proposed for reducing the communication cost by gradient compression or quantization, and the federated versions of adaptive optimizers such as FedAdam are proposed to add more adaptivity, the current federated learning framework still cannot solve the aforementioned challenges all at once. In this paper, we propose a novel communication-efficient adaptive federated learning method (FedCAMS) with theoretical convergence guarantees. We show that in the nonconvex stochastic optimization setting, our proposed FedCAMS achieves the same convergence rate of $O(\frac{1}{\sqrt{TKm}})$ as its non-compressed counterparts. Extensive experiments on various benchmarks verify our theoretical analysis.
翻译:联邦学习是一种机器学习培训模式,使客户能够联合培训模型,而不必分享他们自己的本地数据,然而,在实践中实施联邦学习仍面临许多挑战,例如由于服务器-客户同步的重复和基于SGD的模型更新缺乏适应性,通信管理费用巨大;尽管提出了通过梯度压缩或四分化降低通信成本的各种方法,并提议FedAdam等联合型适应性优化剂增加适应性,但目前的联邦学习框架仍然无法同时解决上述所有挑战。 在本文件中,我们提出了具有理论趋同保证的新的通信高效适应性联邦学习方法(FedCAMS),我们表明,在非康克斯吸附式优化环境下,我们拟议的FedCAMMS实现了与其非压抑性对应方相同的美元(frac{1unsqrt{TKm ⁇ )的趋同率。关于各种基准的广泛实验证实了我们的理论分析。