Explore and Match: A New Paradigm for Temporal Video Grounding with Natural Language

Temporal Video Grounding (TVG) aims to localize time segments in an untrimmed video according to natural language queries. In this work, we present a new paradigm named Explore-and-Match for TVG that seamlessly unifies two streams of TVG methods: proposal-free and proposal-based; the former explores the search space to find segments directly, and the latter matches the predefined proposals with ground truths. To achieve this goal, we view TVG as a set prediction problem and design an end-to-end trainable Language Video Transformer (LVTR) that utilizes the architectural strengths of rich contextualization and parallel decoding for set prediction. The overall training schedule is balanced by two key losses that play different roles, namely temporal localization loss and set guidance loss. These two losses allow each proposal to regress the target segment and identify the target query. More specifically, LVTR first explores the search space to diversify the initial proposals, and then matches the proposals to the corresponding targets to align them in a fine-grained manner. The Explore-and-Match scheme successfully combines the strengths of two complementary methods without encoding prior knowledge (e.g., non-maximum suppression) into the TVG pipeline. As a result, LVTR sets new state-of-the-art results on two TVG benchmarks (ActivityCaptions and Charades-STA) with double the inference speed. Code is available at https://github.com/sangminwoo/Explore-and-Match.