Personalized federated learning considers learning models unique to each client in a heterogeneous network. The resulting client-specific models have been purported to improve metrics such as accuracy, fairness, and robustness in federated networks. However, despite a plethora of work in this area, it remains unclear: (1) which personalization techniques are most effective in various settings, and (2) how important personalization truly is for realistic federated applications. To better answer these questions, we propose Motley, a benchmark for personalized federated learning. Motley consists of a suite of cross-device and cross-silo federated datasets from varied problem domains, as well as thorough evaluation metrics for better understanding the possible impacts of personalization. We establish baselines on the benchmark by comparing a number of representative personalized federated learning methods. These initial results highlight strengths and weaknesses of existing approaches, and raise several open questions for the community. Motley aims to provide a reproducible means with which to advance developments in personalized and heterogeneity-aware federated learning, as well as the related areas of transfer learning, meta-learning, and multi-task learning.
翻译:个人化个人联谊学习认为学习模式是一个多样化网络中每个客户独有的学习模式,由此形成的客户特有模式旨在改进各种衡量标准,如联邦网络的准确性、公平性和稳健性,然而,尽管在这一领域做了大量工作,但仍然不清楚:(1) 哪些个性化技术在各种环境中最为有效,(2) 个性化对于现实的联邦应用来说究竟有多重要;为更好地回答这些问题,我们提议Motley,这是个性化联邦学习的基准。Motley由一系列跨层次和跨层联谊数据集组成,以及全面评价指标,以更好地了解个性化可能产生的影响。我们通过比较一些具有代表性的个人化联邦化学习方法,确定了基准基准。这些初步结果突出了现有方法的长处和短处,并为社区提出了几个开放的问题。Motley的目的是提供一种可再生手段,用以推进个性化和异性-认知的联邦学习,以及相关的转移学习领域。我们通过比较一些具有代表性的个性化、元化和多任务学习方法,从而建立基准。