We design a new sparse projection method for a set of vectors that guarantees a desired average sparsity level measured leveraging the popular Hoyer measure (an affine function of the ratio of the $\ell_1$ and $\ell_2$ norms). Existing approaches either project each vector individually or require the use of a regularization parameter which implicitly maps to the average $\ell_0$-measure of sparsity. Instead, in our approach we set the sparsity level for the whole set explicitly and simultaneously project a group of vectors with the sparsity level of each vector tuned automatically. We show that the computational complexity of our projection operator is linear in the size of the problem. Additionally, we propose a generalization of this projection by replacing the $\ell_1$ norm by its weighted version. We showcase the efficacy of our approach in both supervised and unsupervised learning tasks on image datasets including CIFAR10 and ImageNet. In deep neural network pruning, the sparse models produced by our method on ResNet50 have significantly higher accuracies at corresponding sparsity values compared to existing competitors. In nonnegative matrix factorization, our approach yields competitive reconstruction errors against state-of-the-art algorithms.
翻译:我们为一组矢量设计了一种新的稀少的预测方法,保证以流行的Hoyer测量法(即1美元和2美元标准之比的偏差函数)衡量出理想的平均散射水平; 现有的方法,或者单个地对每个矢量进行预测,或者要求使用一个正规化参数,其中隐含地映射到平均的聚度测量法$/ell_0美元; 相反,在我们的方法中,我们为整个矢量集设定了聚度水平,明确和同时对一组矢量的宽度进行预测; 我们显示,我们预测操作员的计算复杂性在问题大小上是线性的; 此外,我们建议通过以加权版本取代1美元的标准来概括这一预测。 我们展示了我们在包括CIFAR10和图像网络在内的图像数据集上监督和不受监督的学习任务的有效性。 在深层的神经网络中,我们用ResNet50方法生成的稀薄模型在相应的宽度值值值值值值上比现有竞争竞争对手高得多。