The impressive performance exhibited by modern machine learning models hinges on the ability to train such models on a very large amounts of labeled data. However, since access to large volumes of labeled data is often limited or expensive, it is desirable to alleviate this bottleneck by carefully curating the training set. Optimal experimental design is a well-established paradigm for selecting data point to be labeled so to maximally inform the learning process. Unfortunately, classical theory on optimal experimental design focuses on selecting examples in order to learn underparameterized (and thus, non-interpolative) models, while modern machine learning models such as deep neural networks are overparameterized, and oftentimes are trained to be interpolative. As such, classical experimental design methods are not applicable in many modern learning setups. Indeed, the predictive performance of underparameterized models tends to be variance dominated, so classical experimental design focuses on variance reduction, while the predictive performance of overparameterized models can also be, as is shown in this paper, bias dominated or of mixed nature. In this paper we propose a design strategy that is well suited for overparameterized regression and interpolation, and we demonstrate the applicability of our method in the context of deep learning by proposing a new algorithm for single shot deep active learning.
翻译:现代机器学习模型所展示的令人印象深刻的绩效取决于在大量标签数据上培训这类模型的能力。然而,由于获取大量标签数据的机会往往有限或费用昂贵,因此最好通过仔细调整培训集来缓解这一瓶颈。最佳实验设计是选择数据点的既定范例,可以给学习过程以最大的信息。不幸的是,关于最佳实验设计的传统理论侧重于选择实例,以便学习分数不足(因而是非中间的)模型,而现代机器学习模型,例如深神经网络过于分计,而且往往经过训练成为内插。因此,典型的实验设计方法在许多现代学习设置中并不适用。事实上,偏差模型的预测性能往往以差异为主,因此典型的实验设计侧重于缩小差异,而透度过高模型的预测性能也可能如本文所示,偏差主导或混合性质。在本文中,我们提出了一个设计战略,非常适合在深度分析的深度回归和跨演法中,我们通过学习新的深度演算法,来展示了一种超分化的深度演算法。