The deployment of Deep Neural Networks (DNNs) on edge devices is hindered by the substantial gap between performance requirements and available processing power. While recent research has made significant strides in developing pruning methods to build a sparse network for reducing the computing overhead of DNNs, there remains considerable accuracy loss, especially at high pruning ratios. We find that the architectures designed for dense networks by differentiable architecture search methods are ineffective when pruning mechanisms are applied to them. The main reason is that the current method does not support sparse architectures in their search space and uses a search objective that is made for dense networks and does not pay any attention to sparsity. In this paper, we propose a new method to search for sparsity-friendly neural architectures. We do this by adding two new sparse operations to the search space and modifying the search objective. We propose two novel parametric SparseConv and SparseLinear operations in order to expand the search space to include sparse operations. In particular, these operations make a flexible search space due to using sparse parametric versions of linear and convolution operations. The proposed search objective lets us train the architecture based on the sparsity of the search space operations. Quantitative analyses demonstrate that our search architectures outperform those used in the stateof-the-art sparse networks on the CIFAR-10 and ImageNet datasets. In terms of performance and hardware effectiveness, DASS increases the accuracy of the sparse version of MobileNet-v2 from 73.44% to 81.35% (+7.91% improvement) with 3.87x faster inference time.
翻译:深度神经网络(DNN)的部署受到可用处理能力和性能要求之间巨大差距的阻碍。最近的研究在发展剪枝方法以构建稀疏网络以减小DNN的计算负担方面取得了显着进展,但在高剪枝比率下仍存在相当的精度损失。我们发现,当前通过可微分体系结构搜索方法设计的密集神经网络结构,一旦应用剪枝机制,就会变得无效。其中的主要原因在于,当前的方法不支持稀疏结构,并且使用密集网络的搜索目标和不重视稀疏性。在本文中,我们提出了一种新方法来搜索适宜稀疏的神经结构。我们通过在搜索空间中添加两个新的稀疏操作和修改搜索目标来实现此目的。我们提出两种新颖的参数化稀疏卷积和稀疏线性操作,以便通过使用线性和卷积操作的稀疏参数版本扩展搜索空间。所提出的搜索目标使我们可以基于搜索空间操作的稀疏性训练架构。定量分析表明,我们的搜索结构优于用于CIFAR-10和ImageNet数据集的最先进稀疏网络中使用的结构。在性能和硬件效益方面,DASS将MobileNet-v2的稀疏版本的准确性从73.44%提高到81.35%(+7.91%的改进),推理时间快了3.87倍。