Federated Learning (FL) is a distributed machine learning framework to alleviate the data silos, where decentralized clients collaboratively learn a global model without sharing their private data. However, the clients' Non-Independent and Identically Distributed (Non-IID) data negatively affect the trained model, and clients with different numbers of local updates may cause significant gaps to the local gradients in each communication round. In this paper, we propose a Federated Vectorized Averaging (FedVeca) method to address the above problem on Non-IID data. Specifically, we set a novel objective for the global model which is related to the local gradients. The local gradient is defined as a bi-directional vector with step size and direction, where the step size is the number of local updates and the direction is divided into positive and negative according to our definition. In FedVeca, the direction is influenced by the step size, thus we average the bi-directional vectors to reduce the effect of different step sizes. Then, we theoretically analyze the relationship between the step sizes and the global objective, and obtain upper bounds on the step sizes per communication round. Based on the upper bounds, we design an algorithm for the server and the client to adaptively adjusts the step sizes that make the objective close to the optimum. Finally, we conduct experiments on different datasets, models and scenarios by building a prototype system, and the experimental results demonstrate the effectiveness and efficiency of the FedVeca method.
翻译:联邦学习联盟(FL)是一个分布式的机器学习框架,以缓解数据筒仓中的数据问题,分散化的客户在其中合作学习全球模型,而不分享其私人数据。然而,客户的“非独立和同分布(非IID)”数据对经过培训的模式产生了负面影响,而客户的本地更新数量不同,可能会给每个通信回合中的本地梯度带来巨大的差距。在本文中,我们提出了一个联邦矢量递增(FedVeca)方法,以解决上述关于非IID数据的问题。具体地说,我们为与本地梯度相关的全球模型设定了一个新目标。本地梯度的定义是双向向向矢量,其步数为本地更新的数量和方向根据我们的定义分为正负两部分。在FedVeca中,方向受一步大小的影响,因此我们用双向向矢量矢量矢量的矢量来降低不同步骤尺寸的影响。然后,我们从理论上分析与全球梯度和全球梯度相关的模型之间的关系,并获得带有步势大小的双向矢量矢量矢量矢量矢量矢量矢量的双调整,最终的服务器数据调整。最后通过我们按客户的步步数调整了步态的进度的系统,然后通过一步步态调整。