To train robust deep neural networks (DNNs), we systematically study several target modification approaches, which include output regularisation, self and non-self label correction (LC). Three key issues are discovered: (1) Self LC is the most appealing as it exploits its own knowledge and requires no extra models. However, how to automatically decide the trust degree of a learner as training goes is not well answered in the literature. (2) Some methods penalise while the others reward low-entropy predictions, prompting us to ask which one is better. (3) Using the standard training setting, a trained network is of low confidence when severe noise exists, making it hard to leverage its high-entropy self knowledge. To resolve the issue (1), taking two well-accepted propositions--deep neural networks learn meaningful patterns before fitting noise and minimum entropy regularisation principle--we propose a novel end-to-end method named ProSelfLC, which is designed according to learning time and entropy. Specifically, given a data point, we progressively increase trust in its predicted label distribution versus its annotated one if a model has been trained for enough time and the prediction is of low entropy (high confidence). For the issue (2), according to ProSelfLC, we empirically prove that it is better to redefine a meaningful low-entropy status and optimise the learner toward it. This serves as a defence of entropy minimisation. To address the issue (3), we decrease the entropy of self knowledge using a low temperature before exploiting it to correct labels, so that the revised labels redefine a low-entropy target state. We demonstrate the effectiveness of ProSelfLC through extensive experiments in both clean and noisy settings, and on both image and protein datasets. Furthermore, our source code is available at https://github.com/XinshaoAmosWang/ProSelfLC-AT.
翻译:为了培养强大的深层神经网络(DNNS),我们系统地研究若干目标修改方法,其中包括产出规范化、自我和非自我温度校正(LC)等。发现了三个关键问题:(1) 自我测算中心最有吸引力,因为它开发了自己的知识,不需要额外的模型。然而,在培训进行之前如何自动决定学习者的信任度,并没有很好地回答。 (2) 某些方法惩罚低湿度预测,而另一些方法奖励低湿度预测,促使我们询问哪个更好。 (3) 使用标准蛋白质培训设置,当存在严重噪音时,经过培训的网络信心较低,使其难以利用高度自我测算的自我知识。 (1) 采用两个得到良好接受的主张-深层神经网络在适应噪音和最小恒定原则之前学会有意义的模式。 (3) 我们提出一个新的端对端方法名为 ProselfLC,这是根据学习时间和节奏设计设计的。 具体地说,我们通过一个数据点,我们逐渐增加对其预测的标签发行的信任度,而如果一个模型已经训练过足够的自我测算的自我测算,那么,我们开始更精确的自我测算的状态数据。