To effectively exploit the potential of large-scale models, various pre-training strategies supported by massive data from different sources are proposed, including supervised pre-training, weakly-supervised pre-training, and self-supervised pre-training. It has been proved that combining multiple pre-training strategies and data from various modalities/sources can greatly boost the training of large-scale models. However, current works adopt a multi-stage pre-training system, where the complex pipeline may increase the uncertainty and instability of the pre-training. It is thus desirable that these strategies can be integrated in a single-stage manner. In this paper, we first propose a general multi-modal mutual information formula as a unified optimization target and demonstrate that all existing approaches are special cases of our framework. Under this unified perspective, we propose an all-in-one single-stage pre-training approach, named Maximizing Multi-modal Mutual Information Pre-training (M3I Pre-training). Our approach achieves better performance than previous pre-training methods on various vision benchmarks, including ImageNet classification, COCO object detection, LVIS long-tailed object detection, and ADE20k semantic segmentation. Notably, we successfully pre-train a billion-level parameter image backbone and achieve state-of-the-art performance on various benchmarks. Code shall be released.
翻译:为了有效地利用大型模式的潜力,提出了各种培训前战略,并辅之以来自不同来源的大量数据,包括受监督的训练前、受监督的训练前和自监督的训练前和自监督的训练前;已经证明,将多种培训前战略和各种模式/来源的数据结合起来,可以大大促进大型模式的培训;然而,目前的工作采用一个多阶段的训练前系统,复杂的培训前系统可能会增加培训前的不确定性和不稳定性;因此,最好能够将这些战略纳入一个阶段;在本文件中,我们首先提出一个一般的多模式相互信息公式,作为统一优化的目标,并表明所有现有办法都是我们框架的特殊情况;在这一统一的观点下,我们提议一个全阶段的单一培训前方法,名为优化多模式的相互信息前培训(M3I培训前),我们的方法在各种愿景基准方面,包括图像网络分类、COCO物体探测、LVIS前的多模式目标探测和STADE20级基本标准,将成功实现我们10亿个阶段的成绩。