Non-IID data distribution across clients and poisoning attacks are two main challenges in real-world federated learning systems. While both of them have attracted great research interest with specific strategies developed, no known solution manages to address them in a unified framework. To jointly overcome both challenges, we propose SmartFL, a generic approach that optimizes the server-side aggregation process with a small clean server-collected proxy dataset (e.g., around one hundred samples, 0.2% of the dataset) via a subspace training technique. Specifically, the aggregation weight of each participating client at each round is optimized using the server-collected proxy data, which is essentially the optimization of the global model in the convex hull spanned by client models. Since at each round, the number of tunable parameters optimized on the server side equals the number of participating clients (thus independent of the model size), we are able to train a global model with massive parameters using only a small amount of proxy data. We provide theoretical analyses of the convergence and generalization capacity for SmartFL. Empirically, SmartFL achieves state-of-the-art performance on both federated learning with non-IID data distribution and federated learning with malicious clients. The source code will be released.
翻译:在现实世界联合学习系统中,客户之间非IID数据分布和中毒袭击是实际世界联合学习系统中的两个主要挑战。虽然两者都吸引了对所制定的具体战略的极大研究兴趣,但没有任何已知的解决办法能够在统一的框架内解决它们。为了共同克服这两个挑战,我们提议SmartFl, 这是一种通用办法,即通过一个子空间培训技术,优化服务器收集的小型清洁代用服务器数据集(例如,大约100个样本,占数据集0.2%),优化服务器和中毒袭击之间的服务器-侧汇总进程。具体地说,利用服务器收集的代理数据,使每轮参与客户的汇总权重得到优化,这基本上是优化全球模型在客户模型横跨的螺旋壳内优化全球模型。由于每轮,在服务器一侧优化的金枪鱼质参数数量等于参与客户数量(这与模型大小无关),我们能够用少量的代用数据来培训具有庞大参数的全球模型。我们对SmartFLL的趋同和概括能力进行理论分析。SmartFL实现的状态,即SmartFLLF实现由客户组合和Federal-deal II 将使用非数据发布源进行学习。