In this paper, we address the dichotomy between heterogeneous models and simultaneous training in Federated Learning (FL) via a clustering framework. We define a new clustering model for FL based on the (optimal) local models of the users: two users belong to the same cluster if their local models are close; otherwise they belong to different clusters. A standard algorithm for clustered FL is proposed in \cite{ghosh_efficient_2021}, called \texttt{IFCA}, which requires \emph{suitable} initialization and the knowledge of hyper-parameters like the number of clusters (which is often quite difficult to obtain in practical applications) to converge. We propose an improved algorithm, \emph{Successive Refine Federated Clustering Algorithm} (\texttt{SR-FCA}), which removes such restrictive assumptions. \texttt{SR-FCA} treats each user as a singleton cluster as an initialization, and then successively refine the cluster estimation via exploiting similar users belonging to the same cluster. In any intermediate step, \texttt{SR-FCA} uses a robust federated learning algorithm within each cluster to exploit simultaneous training and to correct clustering errors. Furthermore, \texttt{SR-FCA} does not require any \emph{good} initialization (warm start), both in theory and practice. We show that with proper choice of learning rate, \texttt{SR-FCA} incurs arbitrarily small clustering error. Additionally, we validate the performance of our algorithm on standard FL datasets in non-convex problems like neural nets, and we show the benefits of \texttt{SR-FCA} over baselines.
翻译:在本文中, 我们通过集成框架来解决不同模式与联邦学习联盟( FL) 同步培训之间的分化。 我们根据用户的( 最佳) 本地模型定义了一个新的 FL 组合模式: 如果本地模型接近, 两个用户属于同一组群; 否则他们属于不同的组群。 在\cite{ghosh_ valid_ 2021} 中, 提出了分组FL的标准算法, 叫做\ textt{ SR- FCCA}, 需要将每个用户视为单一组群, 然后通过利用属于同一组群的类似用户( 通常很难在实际应用中获得) 来不断改进群集群组群( 通常很难获得) 。 我们提出改进算法的算法, 快速化法的计算法, 向每个中间步骤显示FCFC 的高级变现, 向FCFC 的变现, 向FCFCA 演示的升级 。