The Federated Learning (FL) paradigm is known to face challenges under heterogeneous client data. Local training on non-iid distributed data results in deflected local optimum, which causes the client models drift further away from each other and degrades the aggregated global model's performance. A natural solution is to gather all client data onto the server, such that the server has a global view of the entire data distribution. Unfortunately, this reduces to regular training, which compromises clients' privacy and conflicts with the purpose of FL. In this paper, we put forth an idea to collect and leverage global knowledge on the server without hindering data privacy. We unearth such knowledge from the dynamics of the global model's trajectory. Specifically, we first reserve a short trajectory of global model snapshots on the server. Then, we synthesize a small pseudo dataset such that the model trained on it mimics the dynamics of the reserved global model trajectory. Afterward, the synthesized data is used to help aggregate the deflected clients into the global model. We name our method Dynafed, which enjoys the following advantages: 1) we do not rely on any external on-server dataset, which requires no additional cost for data collection; 2) the pseudo data can be synthesized in early communication rounds, which enables Dynafed to take effect early for boosting the convergence and stabilizing training; 3) the pseudo data only needs to be synthesized once and can be directly utilized on the server to help aggregation in subsequent rounds. Experiments across extensive benchmarks are conducted to showcase the effectiveness of Dynafed. We also provide insights and understanding of the underlying mechanism of our method.
翻译:已知的联邦学习(FL)模式将面临多种客户数据下的挑战。关于非二分配数据的地方培训导致当地最佳数据偏差,导致客户模型相互偏向,降低综合全球模型的性能。自然的解决办法是将所有客户数据收集到服务器上,这样服务器就能对全部数据分布有一个全球视角。不幸的是,这减少为常规培训,这损害了客户的隐私,也与FL的目的相冲突。在本文中,我们提出了一个在服务器上收集和利用全球知识而不妨碍数据隐私的构想。我们从全球模型轨迹的动态中发现这种知识。具体地说,我们首先在服务器上保留一个全球模型截图的短轨。然后,我们合成一个小的假数据,使所培训的模型能够模拟全球模型轨迹的动态。随后,综合数据只能帮助将偏差客户汇总到全球模型中。我们的方法Dynadfed 具有以下优点:(1)我们不依靠任何外部服务器上的精确数据,在服务器的动态中发现这些知识,我们也不依靠任何外部服务器的外观,在服务器上直接推进数据整合数据循环中进行数据整合。我们也可以将数据循环的循环的模拟数据整合。