Deep Ensembles (DE) are a prominent approach for achieving excellent performance on key metrics such as accuracy, calibration, uncertainty estimation, and out-of-distribution detection. However, hardware limitations of real-world systems constrain to smaller ensembles and lower-capacity networks, significantly deteriorating their performance and properties. We introduce Packed-Ensembles (PE), a strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space. We leverage grouped convolutions to parallelize the ensemble into a single shared backbone and forward pass to improve training and inference speeds. PE is designed to operate within the memory limits of a standard neural network. Our extensive research indicates that PE accurately preserves the properties of DE, such as diversity, and performs equally well in terms of accuracy, calibration, out-of-distribution detection, and robustness to distribution shift. We make our code available at https://github.com/ENSTA-U2IS/torch-uncertainty.
翻译:Deep Ensembles是实现关键指标如准确性、校准性、不确定性估计和越界检测卓越性能的突出方法。 然而,现实系统的硬件限制限制了较小的集合和较低容量的网络,极大地降低了它们的性能和属性。我们引入了打包集成(Packed-Ensembles),这是一种通过谨慎调节其编码空间的尺寸来设计和训练轻量级结构集成的策略。我们利用组卷积将集合并行化为单个共享的骨干网络并进行前向传递以提高训练和推理速度。PE旨在在标准神经网络的内存限制内运行。我们广泛的研究表明,PE准确地保留了DE的多样性等特性,并在准确性、校准性、越界检测以及对分布偏移的鲁棒性方面表现出相同的良好表现。我们将我们的代码公开在https://github.com/ENSTA-U2IS/torch-uncertainty。