Video instance shadow detection aims to simultaneously detect, segment, associate, and track paired shadow-object associations in videos. This work has three key contributions to the task. First, we design SSIS-Track, a new framework to extract shadow-object associations in videos with paired tracking and without category specification; especially, we strive to maintain paired tracking even the objects/shadows are temporarily occluded for several frames. Second, we leverage both labeled images and unlabeled videos, and explore temporal coherence by augmenting the tracking ability via an association cycle consistency loss to optimize SSIS-Track's performance. Last, we build $\textit{SOBA-VID}$, a new dataset with 232 unlabeled videos of ${5,863}$ frames for training and 60 labeled videos of ${1,182}$ frames for testing. Experimental results show that SSIS-Track surpasses baselines built from SOTA video tracking and instance-shadow-detection methods by a large margin. In the end, we showcase several video-level applications.
翻译：视频影子检测旨在同时检测、 分段、 关联和跟踪视频中的双向影子对象关联。 这项工作对任务有三大关键贡献。 首先, 我们设计了 SISIS- Track, 这是一个在配对跟踪和无分类规格的视频中提取影子对象关联的新框架; 特别是, 我们努力保持对齐跟踪, 甚至连对象/ 阴影都暂时隐蔽于多个框架。 第二, 我们利用标签图像和未贴标签的视频, 并探索时间一致性, 通过关联周期一致性损失来增强跟踪能力, 优化 SISIS- Track 的性能 。 最后, 我们建造了 $\ textit{ SOBA- VID}$, 一个新的数据集, 包含232个未贴标签的视频 $ 5, 863 美元的培训框架和 60个标签的测试框 ${1, 182 美元。 实验结果显示 SISIS- Track 超过从 SOTA 视频跟踪和实例阴影检测方法中建立的基准。 最后, 我们展示了几个视频级应用程序应用 。
相关内容
微信扫码咨询专知VIP会员