Federated learning methods typically learn a model by iteratively sampling updates from a population of clients. In this work, we explore how the number of clients sampled at each round (the cohort size) impacts the quality of the learned model and the training dynamics of federated learning algorithms. Our work poses three fundamental questions. First, what challenges arise when trying to scale federated learning to larger cohorts? Second, what parallels exist between cohort sizes in federated learning and batch sizes in centralized learning? Last, how can we design federated learning methods that effectively utilize larger cohort sizes? We give partial answers to these questions based on extensive empirical evaluation. Our work highlights a number of challenges stemming from the use of larger cohorts. While some of these (such as generalization issues and diminishing returns) are analogs of large-batch training challenges, others (including training failures and fairness concerns) are unique to federated learning.
翻译:联邦学习方法通常通过对客户群进行迭接抽样更新来学习模式。 在这项工作中,我们探索每轮抽样的客户数量(群体规模)如何影响学习模式的质量和联邦学习算法的培训动态。 我们的工作提出了三个基本问题。 首先,试图将联合学习规模扩大到较大群体时,会遇到哪些挑战?第二,在联合学习中的组群规模与集中学习中的批量规模之间有什么相似之处?最后,我们如何设计能有效利用较大群体群规模的混合学习方法?我们在广泛经验评估的基础上对这些问题给予部分答案。我们的工作突出了使用较大群体组群带来的一些挑战。虽然其中一些(如一般性问题和减少回报)是大型培训挑战的类比,但另一些(包括培训失败和公平问题)是联邦学习所特有的。