A central challenge in training classification models in the real-world federated system is learning with non-IID data. To cope with this, most of the existing works involve enforcing regularization in local optimization or improving the model aggregation scheme at the server. Other works also share public datasets or synthesized samples to supplement the training of under-represented classes or introduce a certain level of personalization. Though effective, they lack a deep understanding of how the data heterogeneity affects each layer of a deep classification model. In this paper, we bridge this gap by performing an experimental analysis of the representations learned by different layers. Our observations are surprising: (1) there exists a greater bias in the classifier than other layers, and (2) the classification performance can be significantly improved by post-calibrating the classifier after federated training. Motivated by the above findings, we propose a novel and simple algorithm called Classifier Calibration with Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated gaussian mixture model. Experimental results demonstrate that CCVR achieves state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10. We hope that our simple yet effective method can shed some light on the future research of federated learning with non-IID data.
翻译:在现实世界联合会系统中,培训分类模式的一个中心挑战是学习非IID数据。为了应对这一挑战,大多数现有工作都涉及加强地方优化的正规化或改进服务器的模型汇总办法。其他工作还共享公共数据集或综合样本,以补充对代表性不足的班级的培训或引入某种程度的个人化。虽然有效,但它们对数据差异性如何影响深度分类模式的每个层面缺乏深刻了解。在本文中,我们通过对不同层次的表述进行实验分析来弥补这一差距。我们的意见令人惊讶:(1) 分类者比其他层次更加偏向于地方,(2) 分类者在经过联邦化培训后进行分类后校准后,分类工作表现可以大大改进。根据上述调查结果,我们提出了一种新颖和简单的算法,称为 " 与虚拟表述模式的分类校准校准 " (CCVR),该算法使用从一个大致的 Gusussian 混合物模型中抽样的虚拟表述来调整分类。实验结果表明,CCVR在分类者中达到了比其他层次更高的水平,比其他层次更加偏差。(2) 分类绩效表现可以通过校准后校准分类方法大大改进。CHR未来学习一些基础的CIRAFIRA-10(C-10号)的简单的学习方法。