Classical results establish that ensembles of small models benefit when predictive diversity is encouraged, through bagging, boosting, and similar. Here we demonstrate that this intuition does not carry over to ensembles of deep neural networks used for classification, and in fact the opposite can be true. Unlike regression models or small (unconfident) classifiers, predictions from large (confident) neural networks concentrate in vertices of the probability simplex. Thus, decorrelating these points necessarily moves the ensemble prediction away from vertices, harming confidence and moving points across decision boundaries. Through large scale experiments, we demonstrate that diversity-encouraging regularizers hurt the performance of high-capacity deep ensembles used for classification. Even more surprisingly, discouraging predictive diversity can be beneficial. Together this work strongly suggests that the best strategy for deep ensembles is utilizing more accurate, but likely less diverse, component models.
翻译:古老的结果证明, 在鼓励预测多样性时, 通过包包、 增强和类似的方式, 小型模型的集合会有利于预测多样性。 我们在这里证明, 这种直觉不会传到用于分类的深神经网络的集合中, 而事实上相反的情况是真实的。 与回归模型或小(不自信的)分类器不同, 大(自信的)神经网络的预测集中在概率简单值的顶部。 因此, 将这些点的装饰必然会转移共同预测, 使其远离悬崖、 损害信心和跨决定边界移动点。 通过大规模实验, 我们证明, 多样性鼓励性规范化者伤害了用于分类的高能力深神经网络的性能。 更令人惊讶的是, 阻止预测性多样性可能是有益的。 这项工作有力地表明, 深海神经网络的最佳战略正在利用更准确但可能更少多样性的成分模型。