While training on samples drawn from independent and identical distribution has been a de facto paradigm for optimizing image classification networks, humans learn new concepts in an easy-to-hard manner and on the selected examples progressively. Driven by this fact, we investigate the training paradigms where the samples are not drawn from independent and identical distribution. We propose a data sampling strategy, named Drop-and-Refresh (DaR), motivated by the learning behaviors of humans that selectively drop easy samples and refresh them only periodically. We show in our experiments that the proposed DaR strategy can maintain (and in many cases improve) the predictive accuracy even when the training cost is reduced by 15% on various datasets (CIFAR 10, CIFAR 100 and ImageNet) and with different backbone architectures (ResNets, DenseNets and MobileNets). Furthermore and perhaps more importantly, we find the ImageNet pre-trained models using our DaR sampling strategy achieves better transferability for the downstream tasks including object detection (+0.3 AP), instance segmentation (+0.3 AP), scene parsing (+0.5 mIoU) and human pose estimation (+0.6 AP). Our investigation encourages people to rethink the connections between the sampling strategy for training and the transferability of its learned features for pre-training ImageNet models.
翻译:虽然从独立和相同的分布中抽取样本的培训是优化图像分类网络的一个事实上的范例,但人类以简单易懂的方式和逐步地从选定的实例中学习新概念。基于这一事实,我们调查了并非从独立和相同的分布中抽取样本的培训模式。我们提出了一个数据抽样战略,名为 " 滴滴和再更新(DaR) ",其动因是人类的学习行为,有选择地投放简单样本并定期刷新样本。我们在实验中显示,拟议的DaR战略可以保持(而且在许多情况下会改进)预测准确性,即使各种数据集(CIFAR 10、CIFAR 100和图像网络)的培训费用减少了15%,并且有不同的主干结构(ResNet、DenseNet和移动网络)。此外,也许更重要的是,我们发现,利用我们的DaR取样战略进行预先培训的模型可以更好地传输下游任务,包括物体探测(+0.3 AP)、例分解(+0.3 AP),现场分析(+0.5 mIOU)和人造型图像分析(+0.6 AP),我们鼓励人们重新思考其取样和图像转换战略之间的连接。