Long iterative training processes for Deep Neural Networks (DNNs) are commonly required to achieve state-of-the-art performance in many computer vision tasks. Importance sampling approaches might play a key role in budgeted training regimes, i.e. when limiting the number of training iterations. These approaches aim at dynamically estimating the importance of each sample to focus on the most relevant and speed up convergence. This work explores this paradigm and how a budget constraint interacts with importance sampling approaches and data augmentation techniques. We show that under budget restrictions, importance sampling approaches do not provide a consistent improvement over uniform sampling. We suggest that, given a specific budget, the best course of action is to disregard the importance and introduce adequate data augmentation; e.g. when reducing the budget to a 30% in CIFAR-10/100, RICAP data augmentation maintains accuracy, while importance sampling does not. We conclude from our work that DNNs under budget restrictions benefit greatly from variety in the training set and that finding the right samples to train on is not the most effective strategy when balancing high performance with low computational requirements. Source code available at https://git.io/JKHa3 .
翻译:为深神经网络(DNN)提供长期的迭代培训程序,对于许多计算机愿景任务实现最先进的业绩而言,通常需要为深神经网络(DNN)提供长期的迭代培训程序,才能在许多计算机愿景任务中实现最先进的业绩; 重要的抽样方法在预算培训制度中可能发挥关键作用,即在限制培训迭代次数时,这些方法的目的是动态地估计每个样本的重要性,以便把重点放在最相关的方面并加快趋同速度。 这项工作探讨了这一模式以及预算制约如何与重要的抽样方法和数据增强技术相互作用。 我们表明,在预算限制下,重要的抽样方法并不比统一抽样提供一致的改进。 我们建议,根据具体预算,最佳的行动方针是忽略重要性,并引入适当的数据增强;例如,在CIFAR-10-100将预算削减到30%时,RICAP数据的增强将保持准确性,而重要性则不那么。 我们从我们的工作得出的结论是,预算限制下的DNN从培训组合中获得很大的好处,而找到正确的培训样品并不是平衡高绩效和低计算要求的最有效战略。