In deep learning, mini-batch training is commonly used to optimize network parameters. However, the traditional mini-batch method may not learn the under-represented samples and complex patterns in the data, leading to a longer time for generalization. To address this problem, a variant of the traditional algorithm has been proposed, which trains the network focusing on mini-batches with high loss. The study evaluates the effectiveness of the proposed training using various deep neural networks trained on three benchmark datasets (CIFAR-10, CIFAR-100, and STL-10). The deep neural networks used in the study are ResNet-18, ResNet-50, Efficient Net B4, EfficientNetV2-S, and MobilenetV3-S. The experimental results showed that the proposed method can significantly improve the test accuracy and speed up the convergence compared to the traditional mini-batch training method. Furthermore, we introduce a hyper-parameter delta ({\delta}) that decides how many mini-batches are considered for training. Experiments on various values of {\delta} found that the performance of the proposed method for smaller {\delta} values generally results in similar test accuracy and faster generalization. We show that the proposed method generalizes in 26.47% less number of epochs than the traditional mini-batch method in EfficientNet-B4 on STL-10. The proposed method also improves the test top-1 accuracy by 7.26% in ResNet-18 on CIFAR-100.
翻译：我们能否从困难的例子中学到更好的东西？ 在深度学习中，通常使用小批量训练来优化网络参数。然而，传统的小批量方法可能无法学习数据中的不足样本和复杂模式，导致更长的普适时间。为了解决这个问题，提出了传统算法的一种变体，该变体针对丢失较高的小批量训练网络。本研究评估了在三个基准数据集（CIFAR-10，CIFAR-100和STL-10）上训练的各种深度神经网络所提出的训练的有效性。本研究中使用的深度神经网络包括ResNet-18，ResNet-50，Efficient Net B4，EfficientNetV2-S和MobilenetV3-S。实验结果表明，所提出的方法可以显着提高测试准确性，并加速收敛速度，相对传统小批量训练方法。此外，我们引入一个超参数delta（$\delta$），该参数决定了要考虑多少个小批量进行训练。对于不同$\delta$值的实验发现，较小的$\delta$值意味着该方法的性能通常是相似的测试精度和更快的普适性。我们展示了所提出的方法在STL-10的EfficientNet-B4上与传统的小批量方法相比，将推广节省26.47％。该方法还可以将CIFAR-100中ResNet-18的测试top-1准确性提高7.26％。
相关内容
微信扫码咨询专知VIP会员