Very deep convolutional neural networks introduced new problems like vanishing gradient and degradation. The recent successful contributions towards solving these problems are Residual and Highway Networks. These networks introduce skip connections that allow the information (from the input or those learned in earlier layers) to flow more into the deeper layers. These very deep models have lead to a considerable decrease in test errors, on benchmarks like ImageNet and COCO. In this paper, we propose the use of exponential linear unit instead of the combination of ReLU and Batch Normalization in Residual Networks. We show that this not only speeds up learning in Residual Networks but also improves the accuracy as the depth increases. It improves the test error on almost all data sets, like CIFAR-10 and CIFAR-100
翻译:极深的革命性神经网络引入了新的问题,比如消失梯度和退化。最近成功地帮助解决这些问题的有遗留物和高速公路网络。这些网络引入了跳过连接,使得信息(从输入或早期层次所学的信息)能够更多地流向更深层层。这些非常深的模型导致图像网络和COCO等基准的测试错误显著减少。在本文中,我们提议使用指数线性单元,而不是将ReLU和残余网络批次正常化结合起来。我们表明,这不仅加快了残余网络的学习,而且随着深度的增加提高了准确性。它改进了几乎所有数据集的测试错误,如CIFAR-10和CIFAR-100。我们建议使用指数线性单元,而不是将RLU和批次正常化结合起来。我们表明,这不仅加快了残余网络的学习速度,而且随着深度的增加,还提高了准确性。它改进了几乎所有数据集(如CIFAR-10和CIFAR-100)的测试错误。