This paper describes our champion solution for the CVPR2022 Generic Event Boundary Captioning (GEBC) competition. GEBC requires the captioning model to have a comprehension of instantaneous status changes around the given video boundary, which makes it much more challenging than conventional video captioning task. In this paper, a Dual-Stream Transformer with improvements on both video content encoding and captions generation is proposed: (1) We utilize three pre-trained models to extract the video features from different granularities. Moreover, we exploit the types of boundary as hints to help the model generate captions. (2) We particularly design an model, termed as Dual-Stream Transformer, to learn discriminative representations for boundary captioning. (3) Towards generating content-relevant and human-like captions, we improve the description quality by designing a word-level ensemble strategy. The promising results on the GEBC test split demonstrate the efficacy of our proposed model.
翻译:本文描述了我们对CVPR2022通用事件边界说明(GEBC)竞赛的首选解决方案。 GEBC要求字幕模型能够理解特定视频边界周围的瞬间状态变化,这比常规视频说明任务更具挑战性。在本文件中,提议采用双层变压器,改进视频内容编码和字幕生成:(1) 我们利用三个预先培训的模型从不同微粒中提取视频特征。此外,我们利用边界类型作为提示帮助模型生成字幕。(2) 我们特别设计了一种模型,称为双层变压器,以学习边界说明的歧视性表述。(3) 为生成内容相关和人性相似的字幕,我们设计了一个字级组合战略,提高了描述质量。GEBC测试的有希望的结果显示我们拟议模型的功效。