This study investigates clustered federated learning (FL), one of the formulations of FL with non-i.i.d. data, where the devices are partitioned into clusters and each cluster optimally fits its data with a localized model. We propose a novel clustered FL framework, which applies a nonconvex penalty to pairwise differences of parameters. This framework can automatically identify clusters without a priori knowledge of the number of clusters and the set of devices in each cluster. To implement the proposed framework, we develop a novel clustered FL method called FPFC. Advancing from the standard ADMM, our method is implemented in parallel, updates only a subset of devices at each communication round, and allows each participating device to perform a variable amount of work. This greatly reduces the communication cost while simultaneously preserving privacy, making it practical for FL. We also propose a new warmup strategy for hyperparameter tuning under FL settings and consider the asynchronous variant of FPFC (asyncFPFC). Theoretically, we provide convergence guarantees of FPFC for general nonconvex losses and establish the statistical convergence rate under a linear model with squared loss. Our extensive experiments demonstrate the advantages of FPFC over existing methods.
翻译:本研究调查了FL的分组化学习(FL),这是FL的组合式学习(FL),这是FL的一种配制,配有非i.i.d.数据,设备被分成组群,每个组群以本地模式优化其数据。我们提议了一个新型分组化FL框架,采用非convex惩罚法对参数差异进行配对。这个框架可以自动识别各组群,而不必事先了解每个组群的组群和成套装置的数量。为了实施拟议的框架,我们开发了一个名为FPFC的新型分组化FL方法。从标准的ADMMM中推进,我们的方法是平行实施的,只更新每轮通信周期的一组装置,并允许每个参与的装置进行可变工作量的工作。这大大降低了通信成本,同时维护隐私,使FL能够实际操作。我们还提出了一个新的超参数调整战略,并考虑FFFPC(asyncFFFFFFFC)的简单变式。理论上,我们为FPFPC的一般非conx损失提供了趋同的趋同性保证,并且确定现有线性实验法的统计优势。