Neural network pruning has been a well-established compression technique to enable deep learning models on resource-constrained devices. The pruned model is usually specialized to meet specific hardware platforms and training tasks (defined as deployment scenarios). However, existing pruning approaches rely heavily on training data to trade off model size, efficiency, and accuracy, which becomes ineffective for federated learning (FL) over distributed and confidential datasets. Moreover, the memory- and compute-intensive pruning process of most existing approaches cannot be handled by most FL devices with resource limitations. In this paper, we develop FedTiny, a novel distributed pruning framework for FL, to obtain specialized tiny models for memory- and computing-constrained participating devices with confidential local data. To alleviate biased pruning due to unseen heterogeneous data over devices, FedTiny introduces an adaptive batch normalization (BN) selection module to adaptively obtain an initially pruned model to fit deployment scenarios. Besides, to further improve the initial pruning, FedTiny develops a lightweight progressive pruning module for local finer pruning under tight memory and computational budgets, where the pruning policy for each layer is gradually determined rather than evaluating the overall deep model structure. Extensive experimental results demonstrate the effectiveness of FedTiny, which outperforms state-of-the-art baseline approaches, especially when compressing deep models to extremely sparse tiny models.
翻译:神经网络修剪一直是一种成熟的压缩技术,使资源限制装置的深学习模式成为资源限制装置。修剪的模型通常专门用来满足具体的硬件平台和培训任务(定义为部署设想方案 ) 。然而,现有的修剪方法严重依赖培训数据来交换模型大小、效率和准确性,这对联邦化学习(FL)分布式和机密数据集而言是无效的。此外,大多数现有办法的记忆和计算密集的修剪过程无法由大多数有资源限制的FL装置处理。在本文中,我们为FDTiny开发了一个新的分布式小程序框架,即FedTiny,这是为FL开发一个新颖的分布式小剪裁框架,以获得用保密的本地数据进行内存和计算机限制的参加装置的微小专门模型。为了减轻由于超常的混杂数据对设备造成的偏差,FedTiny引入了一个适应性批量标准化(BN)选择模块,以适应部署情景的初始修剪剪裁模式。此外,FedTiny开发了一个在紧凑的记忆和计算模型的深度模型下进行地方精细的精细操作的细操作模块模块,以逐步地评估整个实验预算,其中的每一个都显示的精细的实验结果。