Deep neural networks have incredible capacity and expressibility, and can seemingly memorize any training set. This introduces a problem when training in the presence of noisy labels, as the noisy examples cannot be distinguished from clean examples by the end of training. Recent research has dealt with this challenge by utilizing the fact that deep networks seem to memorize clean examples much earlier than noisy examples. Here we report a new empirical result: for each example, when looking at the time it has been memorized by each model in an ensemble of networks, the diversity seen in noisy examples is much larger than the clean examples. We use this observation to develop a new method for noisy labels filtration. The method is based on a statistics of the data, which captures the differences in ensemble learning dynamics between clean and noisy data. We test our method on three tasks: (i) noise amount estimation; (ii) noise filtration; (iii) supervised classification. We show that our method improves over existing baselines in all three tasks using a variety of datasets, noise models, and noise levels. Aside from its improved performance, our method has two other advantages. (i) Simplicity, which implies that no additional hyperparameters are introduced. (ii) Our method is modular: it does not work in an end-to-end fashion, and can therefore be used to clean a dataset for any other future usage.
翻译:深心神经网络具有令人难以置信的能力和可感知性, 并且可以将任何培训组合进行记忆化。 这在使用噪音标签进行训练时提出了一个问题, 因为噪音的例子无法在培训结束时与清洁的例子区分开来。 最近的研究通过利用深心网络似乎比吵闹的例子更早地记住干净的例子来应对这一挑战。 我们在这里报告了一个新的经验结果: 例如,当每个模型在网络的组合中被记忆起来时, 吵闹的例子中看到的多样性比清洁的例子要大得多。 我们利用这一观察来开发一种新的噪音标签过滤方法。 这个方法以数据统计为基础, 收集清洁和吵闹数据之间的混合学习动态差异。 我们用三种任务来测试我们的方法:(一) 噪音量估计;(二) 噪音过滤;(三) 监督分类。 我们用各种数据集、噪音模型和噪音水平来显示我们的方法比所有三项任务的现有基准都好得多。 我们使用这个方法, 我们用这个方法基于数据统计的统计, 而不是用另一种方法。 ( ) 在改进的模型中,我们的方法是另一种方法, 使用另一种方法。