The state of neural network pruning has been noticed to be unclear and even confusing for a while, largely due to "a lack of standardized benchmarks and metrics" [3]. To standardize benchmarks, first, we need to answer: what kind of comparison setup is considered fair? This basic yet crucial question has barely been clarified in the community, unfortunately. Meanwhile, we observe several papers have used (severely) sub-optimal hyper-parameters in pruning experiments, while the reason behind them is also elusive. These sub-optimal hyper-parameters further exacerbate the distorted benchmarks, rendering the state of neural network pruning even more obscure. Two mysteries in pruning represent such a confusing status: the performance-boosting effect of a larger finetuning learning rate, and the no-value argument of inheriting pretrained weights in filter pruning. In this work, we attempt to explain the confusing state of network pruning by demystifying the two mysteries. Specifically, (1) we first clarify the fairness principle in pruning experiments and summarize the widely-used comparison setups; (2) then we unveil the two pruning mysteries and point out the central role of network trainability, which has not been well recognized so far; (3) finally, we conclude the paper and give some concrete suggestions regarding how to calibrate the pruning benchmarks in the future. Code: https://github.com/mingsun-tse/why-the-state-of-pruning-so-confusing.
翻译:神经网络的运行状况在一段时间内被注意到不清楚,甚至甚至令人困惑,这主要是因为“缺乏标准化基准和衡量标准”[3]。为了标准化基准,首先,我们需要回答:什么样的比较设置被认为是公平的?这个基本但至关重要的问题在社区里还没有得到澄清,不幸的是,这个基本但至关重要的问题在社区里还没有得到澄清。与此同时,我们观察到一些论文在运行实验中使用了(严重)亚最佳超参数,而其背后的原因也是难以预料的。这些亚最佳超参数进一步加剧了扭曲的基准,使得神经网络运行状况更加模糊。为了标准化基准,我们需要回答:什么样的比较设置被认为是公平的? 运行中的两个奥秘就代表了这样一个混乱的状况: 更大微调学习率的绩效加速效应, 以及过滤运行过程中继承预加压权的无价值论证。 在这项工作中,我们试图解释网络运行的混乱状态,方法是解开两个奥秘。 具体地说,(1) 我们首先在运行实验中澄清了公平原则,并总结了广泛使用的比较设置;(2) 然后,我们最终揭开了两个核心的角色,我们如何完成。