We address the problem of text-guided video temporal grounding, which aims to identify the time interval of a certain event based on a natural language description. Different from most existing methods that only consider RGB images as visual features, we propose a multi-modal framework to extract complementary information from videos. Specifically, we adopt RGB images for appearance, optical flow for motion, and depth maps for image structure. While RGB images provide abundant visual cues of certain events, the performance may be affected by background clutters. Therefore, we use optical flow to focus on large motion and depth maps to infer the scene configuration when the action is related to objects recognizable with their shapes. To integrate the three modalities more effectively and enable inter-modal learning, we design a dynamic fusion scheme with transformers to model the interactions between modalities. Furthermore, we apply intra-modal self-supervised learning to enhance feature representations across videos for each modality, which also facilitates multi-modal learning. We conduct extensive experiments on the Charades-STA and ActivityNet Captions datasets, and show that the proposed method performs favorably against state-of-the-art approaches.
翻译:我们处理的是文本引导视频时间定位问题,目的是根据自然语言描述确定某一事件的时间间隔。与大多数仅将RGB图像视为视觉特征的现有方法不同,我们提议了一个从视频中提取补充信息的多模式框架。具体地说,我们采用RGB图像作为外观、光学流动和图像结构深度图。虽然RGB图像为某些事件提供了丰富的视觉提示,但性能可能受到背景混杂的影响。因此,我们利用光学流侧重于大型运动和深度地图,以推断与可识别形状的物体有关的场景配置。为了更有效地整合三种模式,并使得能够进行跨模式学习,我们设计了一个与变异器一起的动态融合方案,以模拟各种模式之间的互动。此外,我们采用内部自控自控学习,以加强每种模式视频的特征表现,同时促进多模式学习。我们就Charades-STA和活动网络的数据集进行了广泛的实验,并展示了拟议方法优于状态-艺术方法。