PFGE: Parsimonious Fast Geometric Ensembling of DNNs

Ensemble methods have been widely used to improve the performance of machine learning methods in terms of generalization, while they are hard to use in deep learning systems, as training an ensemble of deep neural networks (DNNs) incurs an extremely higher computational overhead of model training. Recently, advanced techniques such as fast geometric ensembling (FGE) and snapshot ensemble have been proposed. These methods can train the model ensembles in the same time as a single model, thus getting around the hurdle of training time. However, their memory overhead for test-time inference remains much higher than single model based methods. Here we propose a parsimonious FGE (PFGE) that employs a lightweight ensemble of higher-performing DNNs, generated by successively-performed stochastic weight averaging procedures. Experimental results across different advanced DNN architectures on benchmark datasets CIFAR-$\{10,100\}$ and Imagenet, demonstrate that PFGE matches the state-of-the-art FGE method in terms of the generalization error, yet requires only 20% memory overhead for test-time inference. Our code is available at https://github.com/ZJLAB-AMMI/PFGE.