In many machine learning applications where massive and privacy-sensitive data are generated on numerous mobile or IoT devices, collecting data in a centralized location may be prohibitive. Thus, it is increasingly attractive to estimate parameters over mobile or IoT devices while keeping data localized. Such learning setting is known as cross-device federated learning. In this paper, we propose the first theoretically guaranteed algorithms for general minimax problems in the cross-device federated learning setting. Our algorithms require only a fraction of devices in each round of training, which overcomes the difficulty introduced by the low availability of devices. The communication overhead is further reduced by performing multiple local update steps on clients before communication with the server, and global gradient estimates are leveraged to correct the bias in local update directions introduced by data heterogeneity. By developing analyses based on novel potential functions, we establish theoretical convergence guarantees for our algorithms. Experimental results on AUC maximization, robust adversarial network training, and GAN training tasks demonstrate the efficiency of our algorithms.
翻译:在许多机械学习应用程序中,在众多移动或IoT设备上生成了大规模和隐私敏感数据,在集中地点收集数据可能令人望而却步,因此,在保持数据本地化的同时估计移动或IoT设备的参数越来越具有吸引力。这种学习环境被称为跨设备联合学习。在本文中,我们建议对跨设备联合学习环境中的一般小型问题采用第一种理论上有保障的算法。我们的算法在每轮培训中只要求有一小部分设备,这克服了设备低可用率带来的困难。在与服务器通信之前对客户采取多项本地更新步骤,从而进一步降低了通信间接费用。全球梯度估计被利用来纠正数据异质性引入的本地更新方向上的偏差。通过开发基于新潜在功能的分析,我们为我们的算法建立了理论趋同保证。关于AUC最大化的实验结果、强大的对抗网络培训和GAN培训任务证明了我们算法的效率。