The lottery ticket hypothesis suggests that sparse, sub-networks of a given neural network, if initialized properly, can be trained to reach comparable or even better performance to that of the original network. Prior works in lottery tickets have primarily focused on the supervised learning setup, with several papers proposing effective ways of finding "winning tickets" in classification problems. In this paper, we confirm the existence of winning tickets in deep generative models such as GANs and VAEs. We show that the popular iterative magnitude pruning approach (with late rewinding) can be used with generative losses to find the winning tickets. This approach effectively yields tickets with sparsity up to 99% for AutoEncoders, 93% for VAEs and 89% for GANs on CIFAR and Celeb-A datasets. We also demonstrate the transferability of winning tickets across different generative models (GANs and VAEs) sharing the same architecture, suggesting that winning tickets have inductive biases that could help train a wide range of deep generative models. Furthermore, we show the practical benefits of lottery tickets in generative models by detecting tickets at very early stages in training called "early-bird tickets". Through early-bird tickets, we can achieve up to 88% reduction in floating-point operations (FLOPs) and 54% reduction in training time, making it possible to train large-scale generative models over tight resource constraints. These results out-perform existing early pruning methods like SNIP (Lee, Ajanthan, and Torr 2019) and GraSP (Wang, Zhang, and Grosse 2020). Our findings shed light towards existence of proper network initializations that could improve convergence and stability of generative models.
翻译:彩票假设表明,一个特定神经网络的微小小网络,如果能够正确初始化,可以被培训到与原始网络的类似甚至更好的表现。彩票先前的工程主要集中在监督的学习设置上,一些论文提出了在分类问题中找到“获奖票”的有效方法。在本文中,我们确认在诸如GANs和VAEs等深层基因化模型中存在着赢票的可能性。我们表明,流行的迭代级级级拉链(随着较晚的回流)可以用来寻找优胜票。这个方法可以有效地让票达到与原网络的相似或更好的效果。这个方法可以让Auto Encorders的彩票达到99%,VAEs的93%,GANs在CIFAR和Ceereb-A数据集中为89%。我们还可以证明赢票在不同的基因化模型(GANs和VAE)中可以转换成相同的结构,表明胜票的感性偏差可以帮助培养一系列深层次的彩票模型。此外,我们展示了彩票的初等彩票的实际好处是: 彩票的彩票,通过检测模型,让我们的彩票,在早期的彩票中可以实现的彩票中,在ARC模型中可以实现的模型中,在AII的早期测试中,可以实现。