We present a method for matching a text sentence from a given corpus to a given video clip and vice versa. Traditionally video and text matching is done by learning a shared embedding space and the encoding of one modality is independent of the other. In this work, we encode the dataset data in a way that takes into account the query's relevant information. The power of the method is demonstrated to arise from pooling the interaction data between words and frames. Since the encoding of the video clip depends on the sentence compared to it, the representation needs to be recomputed for each potential match. To this end, we propose an efficient shallow neural network. Its training employs a hierarchical triplet loss that is extendable to paragraph/video matching. The method is simple, provides explainability, and achieves state-of-the-art results for both sentence-clip and video-text by a sizable margin across five different datasets: ActivityNet, DiDeMo, YouCook2, MSR-VTT, and LSMDC. We also show that our conditioned representation can be transferred to video-guided machine translation, where we improved the current results on VATEX. Source code is available at https://github.com/AmeenAli/VideoMatch.
翻译:我们提出了一个将文本句从给定内容到给定视频剪辑的匹配方法,反之亦然。 传统的视频和文本匹配是通过学习共享嵌入空间和一种模式的编码独立于另一种模式来完成的。 在这项工作中, 我们以考虑到查询相关信息的方式编码数据集数据。 该方法的力量通过将文字和框架之间的交互数据集中起来而得到证明。 由于视频剪辑的编码取决于该句与该句的比较, 每一个潜在匹配都需要重新配置其表达方式。 为此, 我们建议建立一个高效的浅神经网络。 其培训使用一个可扩展至段落/视频匹配的等级三重损失。 方法简单, 提供解释性, 并且通过五个不同数据集( 活动网、 Diemo、 YouCook 2、 MSR- VTTT 和 LSMDC) 之间的一个可变宽的边距, 显示我们有条件的表达方式可以转换为视频制导导式的机器翻译, 从而改进了 VATEDA/ imevionA.