Structural neural network pruning aims to remove the redundant channels in the deep convolutional neural networks (CNNs) by pruning the filters of less importance to the final output accuracy. To reduce the degradation of performance after pruning, many methods utilize the loss with sparse regularization to produce structured sparsity. In this paper, we analyze these sparsity-training-based methods and find that the regularization of unpruned channels is unnecessary. Moreover, it restricts the network's capacity, which leads to under-fitting. To solve this problem, we propose a novel pruning method, named MaskSparsity, with pruning-aware sparse regularization. MaskSparsity imposes the fine-grained sparse regularization on the specific filters selected by a pruning mask, rather than all the filters of the model. Before the fine-grained sparse regularization of MaskSparity, we can use many methods to get the pruning mask, such as running the global sparse regularization. MaskSparsity achieves 63.03%-FLOPs reduction on ResNet-110 by removing 60.34% of the parameters, with no top-1 accuracy loss on CIFAR-10. On ILSVRC-2012, MaskSparsity reduces more than 51.07% FLOPs on ResNet-50, with only a loss of 0.76% in the top-1 accuracy. The code is released at https://github.com/CASIA-IVA-Lab/MaskSparsity. Moreover, we have integrated the code of MaskSparity into a PyTorch pruning toolkit, EasyPruner, at https://gitee.com/casia_iva_engineer/easypruner.
翻译:结构性神经网络运行的目的是通过剪切对最终输出准确性来说不太重要的过滤器来消除深层神经神经网络中的多余渠道。 为了减少裁剪后性能的退化, 许多方法会利用稀释的正规化方法来产生结构性的垃圾。 在本文中, 我们分析这些基于简单性的培训方法, 并发现对未加工的渠道进行正规化是不必要的。 此外, 它会限制网络的能力, 从而导致配置不足。 为了解决这个问题, 我们建议一种名为 MaskSparity 的新型修剪方法, 且对最终输出准确性不那么重要。 为了减少运行后性能的退化。 MaskSparity 将精细微的稀释性稀释性规范化用于通过修整式遮罩选定的特定过滤器, 而不是所有模型的过滤器。 在精细性地对 MaskS- sparity进行精密性规范之前, 我们可以用许多方法来获取纯度的遮罩, 例如运行全球稀释性规范。 MaskSS- 110 实现60.03%- FLOPs 的削减, 在 AS-RAR- IMFA 上, 在60.34/ RIS 上, 上, 递解 IMIS 上, IMS- 节解 上, IMS- IMS- 上, IMFRS- 节节节节节节节节 上 上 的高级 降为60.34 。