Pruning neural networks has become popular in the last decade when it was shown that a large number of weights can be safely removed from modern neural networks without compromising accuracy. Numerous pruning methods have been proposed since then, each claiming to be better than the previous. Many state-of-the-art (SOTA) techniques today rely on complex pruning methodologies utilizing importance scores, getting feedback through back-propagation or having heuristics-based pruning rules amongst others. We question this pattern of introducing complexity in order to achieve better pruning results. We benchmark these SOTA techniques against Global Magnitude Pruning (Global MP), a naive pruning baseline, to evaluate whether complexity is really needed to achieve higher performance. Global MP ranks weights in order of their magnitudes and prunes the smallest ones. Hence, in its vanilla form, it is one of the simplest pruning techniques. Surprisingly, we find that vanilla Global MP outperforms all the other SOTA techniques and achieves a new SOTA result. It also achieves good performance on FLOPs sparsification, which we find is enhanced, when pruning is conducted in a gradual fashion. We also find that Global MP is generalizable across tasks, datasets and models with superior performance. Moreover, a common issue that many pruning algorithms run into at high sparsity rates, namely, layer-collapse, can be easily fixed in Global MP by setting a minimum threshold of weights to be retained in each layer. Lastly, unlike many other SOTA techniques, Global MP does not require any additional algorithm specific hyper-parameters and is very straightforward to tune and implement. We showcase our findings on various models (WRN-28-8, ResNet-32, ResNet-50, MobileNet-V1 and FastGRNN) and multiple datasets (CIFAR-10, ImageNet and HAR-2). Code is available at https://github.com/manasgupta-1/GlobalMP.
翻译:在过去十年里,神经神经网络变得很受欢迎,当时显示大量重量可以在不降低准确性的情况下安全地从现代神经网络中去除,此后提出了许多修剪方法,每套方法都声称优于以往。今天,许多最先进的工艺(SOTA)技术依靠复杂的修剪方法,利用重要分数,通过反演获得反馈,或采用基于超动的修剪规则等。我们质疑这种引入复杂度的模式,以便实现更直接的调控结果。我们将这些SOTA技术比作全球磁度普鲁宁(Global MP),这是天真性马运基准,以评价是否真的需要复杂性来提高性能。因此,以香草的形式,这是最简单的修剪裁技术之一。我们发现,香草全球马运(SOTA)的多项技术都比其他的直径直径直,并取得了新的SOTA结果。我们用FLOFMS-S-ROTA计算技术也取得了良好的性能表现,而我们又从全球的螺旋上看到一个普通的螺旋和直径直径可操作性模型。