The lottery ticket hypothesis has sparked the rapid development of pruning algorithms that aim to reduce the computational costs associated with deep learning during training and model deployment. Currently, such algorithms are primarily evaluated on imaging data, for which we lack ground truth information and thus the understanding of how sparse lottery tickets could be. To fill this gap, we develop a framework that allows us to plant and hide winning tickets with desirable properties in randomly initialized neural networks. To analyze the ability of state-of-the-art pruning to identify tickets of extreme sparsity, we design and hide such tickets solving four challenging tasks. In extensive experiments, we observe similar trends as in imaging studies, indicating that our framework can provide transferable insights into realistic problems. Additionally, we can now see beyond such relative trends and highlight limitations of current pruning methods. Based on our results, we conclude that the current limitations in ticket sparsity are likely of algorithmic rather than fundamental nature. We anticipate that comparisons to planted tickets will facilitate future developments of efficient pruning algorithms.
翻译:彩票假设引发了快速的修剪算法的发展,目的是降低在培训和模型部署期间深层学习的计算成本。目前,这种算法主要是在成像数据上进行评估,对此我们缺乏地面的真相信息,从而无法理解彩票会多么稀少。为了填补这一空白,我们制定了一个框架,使我们能够在随机初始的神经网络中以理想的属性种植和隐藏入场票。为了分析最先进的修剪算法的能力,以确定极端偏差的门票,我们设计并隐藏这种门票,解决四项具有挑战性的任务。在广泛的实验中,我们观察到类似成像研究中的趋势,我们发现我们的框架可以提供可转移的对现实问题的洞见。此外,我们现在可以看到这种相对趋势之外,并突出当前发盘方法的局限性。根据我们的结果,我们得出结论,目前对进场票的局限性可能是算法性,而不是根本性。我们预计,与人造票的比较将有助于未来高效的修剪算法的发展。