Automated Machine Learning(Auto-ML) pruning methods aim at searching a pruning strategy automatically to reduce the computational complexity of deep Convolutional Neural Networks(deep CNNs). However, some previous work found that the results of many Auto-ML pruning methods cannot even surpass the results of the uniformly pruning method. In this paper, the ineffectiveness of Auto-ML pruning which is caused by unfull and unfair training of the supernet is shown. A deep supernet suffers from unfull training because it contains too many candidates. To overcome the unfull training, a stage-wise pruning(SWP) method is proposed, which splits a deep supernet into several stage-wise supernets to reduce the candidate number and utilize inplace distillation to supervise the stage training. Besides, A wide supernet is hit by unfair training since the sampling probability of each channel is unequal. Therefore, the fullnet and the tinynet are sampled in each training iteration to ensure each channel can be overtrained. Remarkably, the proxy performance of the subnets trained with SWP is closer to the actual performance than that of most of the previous Auto-ML pruning work. Experiments show that SWP achieves the state-of-the-art on both CIFAR-10 and ImageNet under the mobile setting.
翻译:自动机修补(自动- ML) 运行方法旨在自动搜索一个修补策略, 以便自动降低深层进化神经网络( 深度CNN) 的计算复杂性。 但是, 先前的一些工作发现, 许多自动- ML 修补方法的结果甚至不能超过统一修补方法的结果。 本文显示了自动修补( 自动修补) 的无效操作方法, 原因是对超级网进行不完全和不公平的培训。 一个深度的超级网因为包含过多的候选人而受到不完全的培训。 为了克服不完全的培训, 提议了一个阶段错开的运行( SWP) 方法, 将一个深度的超级网分割成几个阶段错开的超级网, 以减少候选数, 并利用本地蒸馏来监督舞台训练。 此外, 一个大的超级网受到不公平的培训的打击, 因为每个频道的取样概率不平等。 因此, 在每个频道的训练中, 全网和小网都受到抽样, 以确保每个频道都可能被过度训练。 值得注意的是, 与SWP 训练过的子网的代理性工作表现接近于S- WP IMAR 之前 和 IMAR 运行 最接近于 的图像 的自动实验状态。