Federated learning (FedL) has emerged as a popular technique for distributing model training over a set of wireless devices, via iterative local updates (at devices) and global aggregations (at the server). In this paper, we develop parallel successive learning (PSL), which expands the FedL architecture along three dimensions: (i) Network, allowing decentralized cooperation among the devices via device-to-device (D2D) communications. (ii) Heterogeneity, interpreted at three levels: (ii-a) Learning: PSL considers heterogeneous number of stochastic gradient descent iterations with different mini-batch sizes at the devices; (ii-b) Data: PSL presumes a dynamic environment with data arrival and departure, where the distributions of local datasets evolve over time, captured via a new metric for model/concept drift. (ii-c) Device: PSL considers devices with different computation and communication capabilities. (iii) Proximity, where devices have different distances to each other and the access point. PSL considers the realistic scenario where global aggregations are conducted with idle times in-between them for resource efficiency improvements, and incorporates data dispersion and model dispersion with local model condensation into FedL. Our analysis sheds light on the notion of cold vs. warmed up models, and model inertia in distributed machine learning. We then propose network-aware dynamic model tracking to optimize the model learning vs. resource efficiency tradeoff, which we show is an NP-hard signomial programming problem. We finally solve this problem through proposing a general optimization solver. Our numerical results reveal new findings on the interdependencies between the idle times in-between the global aggregations, model/concept drift, and D2D cooperation configuration.
翻译:联邦学习(FedL)已经成为通过迭接本地更新(在设备上)和全球聚合(在服务器上)在一组无线设备上分配模式培训的流行技术。 在本文中,我们开发了平行连续学习环境(PSL),在三个层面扩展了FedL架构:(一) 网络,允许通过设备到装置(D2D)通信(D2D),在设备之间分散合作。 (二) 差异性,在三个层面加以解释:(二) 学习:PSL考虑在设备上使用不同微缩的本地更新(在设备上)和全球汇总(在服务器上),通过迭接连更新(PSL),在设备上通过设备到设备到设备到装置上(D2D)通信(D2D)通讯(D2D),使设备能够分散合作。 (二) 差异性,在三个层次上解释:(二) 学习:PSLSL考虑在设备与其他设备之间有不同动态距离的易变梯梯脱落梯落差的迭迭迭迭迭迭; (二) 数据:PSL) 最终设想,在模型上进行全球聚合模型,在数据流运行中,在数据流中,在数据流流流流流流流中,在数据流转动中,在数据流流流流流流流转中,在数据流转中,在数据流转到流转,在数据流转到我们内部流转中显示中,在资源流流,在资源流转中将数据流数据流流流流流数据流数据流流,在资源流,在数据流流中显示到我们对流,在资源流流数据流数据流数据流数据流数据流数据流数据流数据流数据流数据流数据流数据流到我们向中显示到我们向中显示到我们向中,在数据流到我们向中显示中显示到我们向中显示到我们向中显示到我们向中将数据流到我们向中显示中显示到我们向中显示中显示中显示中显示到我们向中显示。