We propose an asymptotic framework to analyze the performance of (personalized) federated learning algorithms. In this new framework, we formulate federated learning as a multi-criterion objective, where the goal is to minimize each client's loss using information from all of the clients. We analyze a linear regression model where, for a given client, we may theoretically compare the performance of various algorithms in the high-dimensional asymptotic limit. This asymptotic multi-criterion approach naturally models the high-dimensional, many-device nature of federated learning. These tools make fairly precise predictions about the benefits of personalization and information sharing in federated scenarios -- at least in our (stylized) model -- including that Federated Averaging with simple client fine-tuning achieves the same asymptotic risk as the more intricate meta-learning and proximal-regularized approaches and outperforming Federated Averaging without personalization. We evaluate these predictions on federated versions of the EMNIST, CIFAR-100, Shakespeare, and Stack Overflow datasets, where the experiments corroborate the theoretical predictions, suggesting such frameworks may provide a useful guide to practical algorithmic development.
翻译:我们提出了一个分析(个性化)联谊学习算法绩效的无线回归框架。在这个新框架内,我们将联谊学习作为一个多标准目标,目标是利用所有客户的信息最大限度地减少每个客户的损失。我们分析了一个线性回归模型,对于一个特定客户,我们可从理论上比较高维无症状极限中各种算法的绩效。这种无线性多级标准方法自然地模拟了联谊学习的高维性、多端性质。这些工具对在联谊情景中的个人化和信息共享的好处 -- -- 至少在我们的(银化)模型中 -- -- 做出了相当精确的预测,包括采用简单客户微调的联谊变异能够实现与更复杂的元学习和Proximal-常规化方法相同的风险,并在没有个性化的情况下超越了联谊。我们评估了这些关于EMNIST、CIFAR-100、莎士兰和Stack 过度流数据共享版本的预测。这些预测对在联邦化情景中的个人化和信息共享的好处 -- -- 至少在我们的(银化)模型模式中 -- -- -- 包括以简单客户微调化方式调整的混合算算算法提供了实用的理论性算法框架。