Federated learning enables resource-constrained edge compute devices, such as mobile phones and IoT devices, to learn a shared model for prediction, while keeping the training data local. This decentralized approach to train models provides privacy, security, regulatory and economic benefits. In this work, we focus on the statistical challenge of federated learning when local data is non-IID. We first show that the accuracy of federated learning reduces significantly, by up to 55% for neural networks trained for highly skewed non-IID data, where each client device trains only on a single class of data. We further show that this accuracy reduction can be explained by the weight divergence, which can be quantified by the earth mover's distance (EMD) between the distribution over classes on each device and the population distribution. As a solution, we propose a strategy to improve training on non-IID data by creating a small subset of data which is globally shared between all the edge devices. Experiments show that accuracy can be increased by 30% for the CIFAR-10 dataset with only 5% globally shared data.
翻译:联邦学习能够让资源受限制的边缘计算设备,如移动电话和IoT设备等,学习一个共同的预测模型,同时保持培训数据本地化。这种分散化的模型培训方法提供了隐私、安全、监管和经济效益。在这项工作中,当当地数据为非IID时,我们集中关注联合学习的统计挑战。我们首先表明,联邦学习的准确性显著下降,对于受过高度扭曲的非IID数据训练的神经网络来说,下降率高达55%,而每个客户设备只能用单类数据进行训练。我们进一步表明,这种准确性下降可以由重量差异来解释,这种差异可以用地球移动器对每个设备班级的分配与人口分布之间的距离来量化。作为一种解决办法,我们提议了一项战略,通过建立一小部分数据来改进关于非IID数据的培训,这些数据在全球所有边缘设备之间共享。实验显示,只有5%的全球共享数据,CIFAR-10数据集的准确性可以提高30%。