The training data distribution is often biased towards objects in certain orientations and illumination conditions. While humans have a remarkable capability of recognizing objects in out-of-distribution (OoD) orientations and illuminations, Deep Neural Networks (DNNs) severely suffer in this case, even when large amounts of training examples are available. In this paper, we investigate three different approaches to improve DNNs in recognizing objects in OoD orientations and illuminations. Namely, these are (i) training much longer after convergence of the in-distribution (InD) validation accuracy, i.e., late-stopping, (ii) tuning the momentum parameter of the batch normalization layers, and (iii) enforcing invariance of the neural activity in an intermediate layer to orientation and illumination conditions. Each of these approaches substantially improves the DNN's OoD accuracy (more than 20% in some cases). We report results in four datasets: two datasets are modified from the MNIST and iLab datasets, and the other two are novel (one of 3D rendered cars and another of objects taken from various controlled orientations and illumination conditions). These datasets allow to study the effects of different amounts of bias and are challenging as DNNs perform poorly in OoD conditions. Finally, we demonstrate that even though the three approaches focus on different aspects of DNNs, they all tend to lead to the same underlying neural mechanism to enable OoD accuracy gains --individual neurons in the intermediate layers become more selective to a category and also invariant to OoD orientations and illuminations. We anticipate this study to be a basis for further improvement of deep neural networks' OoD generalization performance, which is highly demanded to achieve safe and fair AI applications.
翻译:培训数据分布往往偏向某些方向和光化条件下的物体。虽然人类具有显著的能力,能够识别分配外(OoD)方向和光化过程中的物体,但深神经网络(DNNS)在此情况下遭受严重痛苦,即使有大量培训实例。在本文件中,我们调查了三种不同的方法来改进DNS在识别OoD方向和光化中的物体方面的精确度。我们报告的数据有四个数据集:(一) 在分配内(InD)验证准确度趋同之后,培训时间长得多,即,停止后,(二)调整批次正常化层(OOD)方向和光化灯光化的物体,但深层神经网络(D)方向的亮度参数也非常差,而ONW的精确性能也更低,而DOO的精确性能也更低,而DOI的精确性能也更低,而OOI的精确性能也更低。