Online action detection is the task of predicting the action as soon as it happens in a streaming video. A major challenge is that the model does not have access to the future and has to solely rely on the history, i.e., the frames observed so far, to make predictions. It is therefore important to accentuate parts of the history that are more informative to the prediction of the current frame. We present GateHUB, Gated History Unit with Background Suppression, that comprises a novel position-guided gated cross-attention mechanism to enhance or suppress parts of the history as per how informative they are for current frame prediction. GateHUB further proposes Future-augmented History (FaH) to make history features more informative by using subsequently observed frames when available. In a single unified framework, GateHUB integrates the transformer's ability of long-range temporal modeling and the recurrent model's capacity to selectively encode relevant information. GateHUB also introduces a background suppression objective to further mitigate false positive background frames that closely resemble the action frames. Extensive validation on three benchmark datasets, THUMOS, TVSeries, and HDD, demonstrates that GateHUB significantly outperforms all existing methods and is also more efficient than the existing best work. Furthermore, a flow-free version of GateHUB is able to achieve higher or close accuracy at 2.8x higher frame rate compared to all existing methods that require both RGB and optical flow information for prediction.
翻译:在线行动探测是一项任务,即一旦在流动视频中发生,即对行动进行预测。一个重大挑战是模型无法进入未来,只能依靠历史,即迄今为止所观察到的框架,作出预测。因此,必须突出历史中对当前框架预测更为丰富的部分信息。我们介绍GateHUB,GateHUB,GateTHUB,带有《背景限制》的GateTHUB,它包含一个新颖的定位制导的封闭式交叉注意机制,目的是根据当前框架预测的信息,加强或压制历史的某些部分。GateHUB进一步提议未来启动历史,以便通过使用以后所观察到的框架使历史特征更加丰富信息。在一个单一的统一框架内,GateHUB将变异器的长程模型能力与经常性模型对相关信息进行选择性编码的能力结合起来。GateHUB还引入了背景抑制目标,以进一步减少与行动框架相近的虚假正面背景框架。在三个基准数据集上进行广泛的验证,THUMOOS, TVSSRESrimes, 和GHDDF的更精确的精确度, 也大大地要求所有现有的GOLF 和GUB 的流程框架, 都比目前更精确的方法都更精确的系统,要比现在更精确的流程和HDDDDDDDF 显示所有。