Real-world videos contain many complex actions with inherent relationships between action classes. In this work, we propose an attention-based architecture that models these action relationships for the task of temporal action localization in untrimmed videos. As opposed to previous works that leverage video-level co-occurrence of actions, we distinguish the relationships between actions that occur at the same time-step and actions that occur at different time-steps (i.e. those which precede or follow each other). We define these distinct relationships as action dependencies. We propose to improve action localization performance by modeling these action dependencies in a novel attention-based Multi-Label Action Dependency (MLAD)layer. The MLAD layer consists of two branches: a Co-occurrence Dependency Branch and a Temporal Dependency Branch to model co-occurrence action dependencies and temporal action dependencies, respectively. We observe that existing metrics used for multi-label classification do not explicitly measure how well action dependencies are modeled, therefore, we propose novel metrics that consider both co-occurrence and temporal dependencies between action classes. Through empirical evaluation and extensive analysis, we show improved performance over state-of-the-art methods on multi-label action localization benchmarks(MultiTHUMOS and Charades) in terms of f-mAP and our proposed metric.
翻译:现实世界视频包含许多复杂的行动,具有行动类别之间固有的内在关系。 在这项工作中,我们建议一个基于关注的架构,在未剪动的视频中为时间行动定位任务模拟这些行动关系。与以前利用视频级行动共同发生的工作相比,我们区分同时发生的行动和在不同时间步骤(即先行或后行)发生的行动之间的关系。我们将这些不同关系定义为行动依赖关系。我们建议通过在新的关注型多周期行动依赖(MLAD)中建模这些行动依赖关系来改进行动本地化绩效。与以前利用视频级共同行动共同发生的工作不同,我们区分同时发生的行动和在不同时间步骤(即先行或后行的行动)发生的行动之间的关系。我们发现,目前用于多标签分类的衡量标准并不明确衡量行动依赖性,因此,我们提出了新的衡量标准,其中既考虑关注关联性,又考虑不同时间依赖性的多周期(MLADAD) 。通过对业绩和多年度评估方法进行我们之间业绩评估并展示行动基准。