Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain. Although VideoMAE has trained a robust ViT from limited data, its low-level reconstruction poses convergence difficulties and conflicts with high-level cross-modal alignment. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. To increase data efficiency, we mask out most of the low-semantics video tokens, but selectively align the unmasked tokens with IFM, which serves as the UnMasked Teacher (UMT). By providing semantic guidance, our method enables faster convergence and multimodal friendliness. With a progressive pre-training framework, our model can handle various tasks including scene-related, temporal-related, and complex video-language understanding. Using only public sources for pre-training in 6 days on 32 A100 GPUs, our scratch-built ViT-L/16 achieves state-of-the-art performances on various video tasks. The code and models will be released at https://github.com/OpenGVLab/unmasked_teacher.
翻译:视频特征模型(VFM)由于高计算成本和数据稀缺性,受到了限制。先前的VFM依赖于图像特征模型(IFM),后者在转移到视频领域时面临着挑战。虽然 VideoMAE 通过有限的数据训练出了一个强大的 ViT,但其低级重建会导致收敛困难,以及与高级跨模态对齐产生冲突。本文提出了一种训练高效的方法,用于集成现有方法的优点,以生成具有时态敏感性的视频特征模型。为了提高数据效率,我们掩盖了大部分低语义的视频标记,但是又有选择性地将未掩盖的标记与IFM对齐,后者作为UnMasked Teacher (UMT)。通过提供语义指导,我们的方法使得模型具备更快的收敛速度和多模态友好性。通过逐步预训练框架,我们的模型可以处理各种任务,包括与场景相关、与时间相关和复杂的视频语言理解。在仅用公共数据进行 6 天、在 32 A100 GPU 上的预训练情况下,我们基于零的 ViT-L/16 在各种视频任务上取得了最先进的性能。可以在 https://github.com/OpenGVLab/unmasked_teacher 上发布代码和模型。