Deep neural networks (DNNs) are effective in solving many real-world problems. Larger DNN models usually exhibit better quality (e.g., accuracy) but their excessive computation results in long training and inference time. Model sparsification can reduce the computation and memory cost while maintaining model quality. Most existing sparsification algorithms unidirectionally remove weights, while others randomly or greedily explore a small subset of weights in each layer. The inefficiency of the algorithms reduces the achievable sparsity level. In addition, many algorithms still require pre-trained dense models and thus suffer from large memory footprint and long training time. In this paper, we propose a novel scheduled grow-and-prune (GaP) methodology without pre-training the dense models. It addresses the shortcomings of the previous works by repeatedly growing a subset of layers to dense and then pruning back to sparse after some training. Experiments have shown that such models can match or beat the quality of highly optimized dense models at 80% sparsity on a variety of tasks, such as image classification, objective detection, 3D object part segmentation, and translation. They also outperform other state-of-the-art (SOTA) pruning methods, including pruning from pre-trained dense models. As an example, a 90% sparse ResNet-50 obtained via GaP achieves 77.9% top-1 accuracy on ImageNet, improving the SOTA results by 1.5%.
翻译:深心内网( DNN) 能够有效解决许多真实世界的问题。 大的 DNN 模型通常显示质量更高( 准确性), 但其过量计算导致长期培训和推断时间过长。 模型垃圾化可以降低计算和记忆成本, 同时保持模型质量。 大部分现有的垃圾化算法单向去除重量, 而另一些则随机或贪婪地探索每一层的一小部分重量。 算法效率低, 降低了可实现的宽度水平。 此外, 许多算法仍然需要经过预先训练的密集模型, 并因此遭受大量记忆足迹和长时间的培训时间。 在本文件中, 我们提出一个新的预定的生长- 发源( GaP) 方法, 无需对密度模型进行预先训练。 它解决了以前工作的缺点, 反复增加一组层到密度, 然后在一些培训后又又又回溯到稀薄。 实验显示, 这种模型可以匹配或击败高精密密度模型的质量, 80 % 的深度模型在各种任务上, 例如图像分类、 客观检测、 3D- 目标- TA- train- trainal- train the a ex- presmission- pexfain resmilling prestical resmation restical ex