Overfitting is a common problem in machine learning, which means the model too closely fits the training data while performing poorly in the test data. Among various methods of coping with overfitting, dropout is one of the representative ways. From randomly dropping neurons to dropping neural structures, dropout has achieved great success in improving model performances. Although various dropout methods have been designed and widely applied in past years, their effectiveness, application scenarios, and contributions have not been comprehensively summarized and empirically compared by far. It is the right time to make a comprehensive survey. In this paper, we systematically review previous dropout methods and classify them into three major categories according to the stage where dropout operation is performed. Specifically, more than seventy dropout methods published in top AI conferences or journals (e.g., TKDE, KDD, TheWebConf, SIGIR) are involved. The designed taxonomy is easy to understand and capable of including new dropout methods. Then, we further discuss their application scenarios, connections, and contributions. To verify the effectiveness of distinct dropout methods, extensive experiments are conducted on recommendation scenarios with abundant heterogeneous information. Finally, we propose some open problems and potential research directions about dropout that worth to be further explored.
翻译:机械学习是一个常见的问题,这意味着模型在测试数据中表现不佳时与培训数据过于吻合,在测试数据表现不佳时与培训数据过于吻合。在各种处理过度装配的方法中,辍学是具有代表性的方法之一。从随机投出神经元到投出神经结构,辍学在改善模型性能方面取得了巨大成功。尽管过去几年设计了各种辍学方法,并广泛应用了这些方法,但其有效性、应用情景和贡献并没有得到全面总结和经验方面的比较。现在是进行全面调查的适当时机。在本文中,我们系统地审查以往的辍学方法,并根据实施辍学作业的阶段将其分为三大类。具体地说,在最高AI会议或期刊(例如,TKDE、KDD、ThebConf、SIGIR)上公布的70多种辍学方法都涉及到了。设计中的分类方法很容易理解,而且能够包括新的辍学方法。随后,我们进一步讨论了它们的应用情景、联系和贡献。为了核实不同的辍学方法的有效性,我们用大量不同的信息对建议性假设进行了广泛的实验。最后,我们提出了值得进一步探讨的一些公开的问题和潜在的研究方向。