Recent works in video prediction have mainly focused on passive forecasting and low-level action-conditional prediction, which sidesteps the learning of interaction between agents and objects. We introduce the task of semantic action-conditional video prediction, which uses semantic action labels to describe those interactions and can be regarded as an inverse problem of action recognition. The challenge of this new task primarily lies in how to effectively inform the model of semantic action information. Inspired by the idea of Mixture of Experts, we embody each abstract label by a structured combination of various visual concept learners and propose a novel video prediction model, Modular Action Concept Network (MAC). Our method is evaluated on two newly designed synthetic datasets, CLEVR-Building-Blocks and Sapien-Kitchen, and one real-world dataset called Tower-Creation. Extensive experiments demonstrate that MAC can correctly condition on given instructions and generate corresponding future frames without need of bounding boxes. We further show that the trained model can make out-of-distribution generalization, be quickly adapted to new object categories and exploit its learnt features for object detection, showing the progression towards higher-level cognitive abilities.
翻译:视频预测的近期工作主要侧重于被动预测和低水平行动条件预测,这阻碍了对物剂和物体之间互动的学习。我们引入了语义行动条件视频预测任务,使用语义行动标志描述这些互动,可被视为反向行动识别问题。这一新任务的挑战主要在于如何有效地为语义行动信息模型提供信息。受专家混合理念的启发,我们通过由各种视觉概念学习者组成的结构化组合,体现了每个抽象标签,并提出了一个新的视频预测模型,即“模子行动概念网络 ” (MAC)。我们的方法是在两个新设计的合成数据集(CLEVR-Building-Blocks和Sapien-Kitchen)和一个称为“Tow-Cregation”的现实世界数据集(Tow-Cregation)上进行评估的。广泛的实验表明,MAC可以正确地为给定指示提供条件,并生成相应的未来框架,而不需要捆绑框。我们进一步表明,经过培训的模型可以进行外向外扩展,迅速适应新的对象类别,并利用其学习的特性来探测物体的能力。