Ensembles of CNN models trained with different seeds (also known as Deep Ensembles) are known to achieve superior performance over a single copy of the CNN. Neural Ensemble Search (NES) can further boost performance by adding architectural diversity. However, the scope of NES remains prohibitive under limited computational resources. In this work, we extend NES to multi-headed ensembles, which consist of a shared backbone attached to multiple prediction heads. Unlike Deep Ensembles, these multi-headed ensembles can be trained end to end, which enables us to leverage one-shot NAS methods to optimize an ensemble objective. With extensive empirical evaluations, we demonstrate that multi-headed ensemble search finds robust ensembles 3 times faster, while having comparable performance to other ensemble search methods, in both predictive performance and uncertainty calibration.
翻译:由不同种子(又称深团)培训的CNN模型组合已知能比CNN的单一版本取得优异性能。神经聚合搜索(NES)可以通过增加建筑多样性进一步提升性能。然而,在有限的计算资源下,NES的范围仍然令人望而却步。在这项工作中,我们将NES扩大到多头组合,由多个预测头的共脊组成。与深团不同的是,这些多头组合可以培训结束,从而使我们能够利用一发NAS方法优化共性目标。通过广泛的经验评估,我们证明多头共振搜索发现强型集合速度为3倍,同时在预测性能和不确定性校准方面与其他共载搜索方法具有相似性。