Efficient deployment of deep neural networks across many devices and resource constraints, especially on edge devices, is one of the most challenging problems in the presence of data-privacy preservation issues. Conventional approaches have evolved to either improve a single global model while keeping each local training data decentralized (i.e., data-heterogeneity) or to train a once-for-all network that supports diverse architectural settings to address heterogeneous systems equipped with different computational capabilities (i.e., model-heterogeneity). However, little research has considered both directions simultaneously. In this work, we propose a novel framework to consider both scenarios, namely Federation of Supernet Training (FedSup), where clients send and receive a supernet whereby it contains all possible architectures sampled from itself. It is inspired by how averaging parameters in the model aggregation stage of Federated Learning (FL) is similar to weight-sharing in supernet training. Specifically, in the FedSup framework, a weight-sharing approach widely used in the training single shot model is combined with the averaging of Federated Learning (FedAvg). Under our framework, we present an efficient algorithm (E-FedSup) by sending the sub-model to clients in the broadcast stage for reducing communication costs and training overhead. We demonstrate several strategies to enhance supernet training in the FL environment and conduct extensive empirical evaluations. The resulting framework is shown to pave the way for the robustness of both data- and model-heterogeneity on several standard benchmarks.
翻译:在存在数据隐私保护问题的情况下,高效部署跨越许多装置和资源制约的深层神经网络,特别是边缘装置,是最具挑战性的问题之一。常规方法已经演变为改进单一全球模型,同时保持每个地方培训数据分散化(即数据异质性),或培训一个一劳永逸的网络,以支持各种建筑环境,处理具有不同计算能力(即模型异质性)的不同系统。然而,研究很少同时考虑两个方向。在这项工作中,我们提议了一个新的框架来考虑两种设想方案,即超级网络培训联合会(FedSup),客户发送和接收一个超级网络,其中包含所有可能的样本结构。它受到联邦学习(FLF)模型综合阶段平均参数如何与超级网络培训中的权重分担相似。具体地说,在FedSup框架内,培训中广泛使用的权重分担方法与联邦学习平均法(FedAvg)相结合。在我们的框架内,我们展示了一种高效的、高超清晰度的通信成本,在交付的F-FS级培训阶段,我们展示了一种高效的、高清晰度的、高清晰度的计算方法,我们展示了用于交付的、高清晰的计算机操作环境。