In federated learning (FL), the objective of collaboratively learning a global model through aggregation of model updates across devices tends to oppose the goal of personalization via local information. In this work, we calibrate this tradeoff in a quantitative manner through a multi-criterion optimization-based framework, which we cast as a constrained program: the objective for a device is its local objective, which it seeks to minimize while satisfying nonlinear constraints that quantify the proximity between the local and the global model. By considering the Lagrangian relaxation of this problem, we develop an algorithm that allows each node to minimize its local component of Lagrangian through queries to a first-order gradient oracle. Then, the server executes Lagrange multiplier ascent steps followed by a Lagrange multiplier-weighted averaging step. We call this instantiation of the primal-dual method Federated Learning Beyond Consensus ($\texttt{FedBC}$). Theoretically, we establish that $\texttt{FedBC}$ converges to a first-order stationary point at rates that matches the state of the art, up to an additional error term that depends on the tolerance parameter that arises due to the proximity constraints. Overall, the analysis is a novel characterization of primal-dual methods applied to non-convex saddle point problems with nonlinear constraints. Finally, we demonstrate that $\texttt{FedBC}$ balances the global and local model test accuracy metrics across a suite of datasets (Synthetic, MNIST, CIFAR-10, Shakespeare), achieving competitive performance with the state of the art.
翻译:在联合学习(FL)中,通过跨设备的模型更新汇总合作学习全球模型的目标往往反对通过本地信息实现个性化的目标。在这项工作中,我们通过一个多标准优化框架,以量化的方式校准这种权衡,我们将此设定为一个受限程序:设备的目标是其本地目标,它试图在满足非线性限制,以量化本地和全球模型之间的距离的同时将其最小化。考虑到Lagrangian对该问题的放松,我们开发了一种算法,允许每个节点通过查询一级精度梯度或触角来尽量减少其Lagrangian的本地组成部分。然后,服务器将拉格朗基乘数乘数增量,然后是拉格兰基增量平均步数框架。我们称之为该设备的目标就是其本地目标,同时满足非线性能限制的非线性能限制非线性(Frickralalalalalal),我们确定该模型与第一个端点相匹配的固定点(Rabinal-ral-rass), 直径Seral-lax 度测试术语取决于最终的度限制度测试方法。