Mixup is a data augmentation technique that relies on training using random convex combinations of data points and their labels. In recent years, Mixup has become a standard primitive used in the training of state-of-the-art image classification models due to its demonstrated benefits over empirical risk minimization with regards to generalization and robustness. In this work, we try to explain some of this success from a feature learning perspective. We focus our attention on classification problems in which each class may have multiple associated features (or views) that can be used to predict the class correctly. Our main theoretical results demonstrate that, for a non-trivial class of data distributions with two features per class, training a 2-layer convolutional network using empirical risk minimization can lead to learning only one feature for almost all classes while training with a specific instantiation of Mixup succeeds in learning both features for every class. We also show empirically that these theoretical insights extend to the practical settings of image benchmarks modified to have additional synthetic features.
翻译:混合是一种数据增强技术,它依赖于使用随机混凝土组合数据点及其标签的培训。近年来,混合已成为用于培训最新图像分类模型的标准原始方法,因为事实证明,在一般化和稳健性方面,在经验风险最小化方面,混合的好处大于经验风险最小化。在这项工作中,我们试图从特征学习的角度来解释其中的一些成功。我们把注意力集中在分类问题上,其中每个班级可能有多种相关特征(或视图),可用于正确预测班级。我们的主要理论结果显示,对于每班有两个特征的非三级数据分布班级,使用实验风险最小化的经验风险培训二层共生网络只能导致几乎所有班级只学习一个特征,而以特定的即时化混合法培训在学习每个班的两种特征方面都取得成功。我们还从经验上表明,这些理论洞察力可以延伸到为额外合成特征而修改的图像基准的实用环境。