On Mixup Regularization

Mixup is a data augmentation technique that creates new examples as convex combinationsof training points and labels. This simple technique has empirically shown to improvethe accuracy of many state-of-the-art models in different settings and applications, butthe reasons behind this empirical success remain poorly understood. In this paper wetake a substantial step in explaining the theoretical foundations of Mixup, by clarifyingits regularization effects. We show that Mixup can be interpreted as standard empiricalrisk minimization estimator subject to a combination of data transformation and randomperturbation of the transformed data. We gain two core insights from this new interpretation.First, the data transformation suggests that, at test time, a model trained with Mixup shouldalso be applied to transformed data, a one-line change in code that we show empirically toimprove both accuracy and calibration of the prediction. Second, we show how the randomperturbation of the new interpretation of Mixup induces multiple known regularizationschemes, including label smoothing and reduction of the Lipschitz constant of the estimator.These schemes interact synergistically with each other, resulting in a self calibrated andeffective regularization effect that prevents overfitting and overconfident predictions. Wecorroborate our theoretical analysis with experiments that support our conclusions.