Contrastive learning has nearly closed the gap between supervised and self-supervised learning of image representations, and has also been explored for videos. However, prior work on contrastive learning for video data has not explored the effect of explicitly encouraging the features to be distinct across the temporal dimension. We develop a new temporal contrastive learning framework consisting of two novel losses to improve upon existing contrastive self-supervised video representation learning methods. The local-local temporal contrastive loss adds the task of discriminating between non-overlapping clips from the same video, whereas the global-local temporal contrastive aims to discriminate between timesteps of the feature map of an input clip in order to increase the temporal diversity of the learned features. Our proposed temporal contrastive learning framework achieves significant improvement over the state-of-the-art results in various downstream video understanding tasks such as action recognition, limited-label action classification, and nearest-neighbor video retrieval on multiple video datasets and backbones. We also demonstrate significant improvement in fine-grained action classification for visually similar classes. With the commonly used 3D ResNet-18 architecture, we achieve 82.4% (+5.1% increase over the previous best) top-1 accuracy on UCF101 and 52.9% (+5.4% increase) on HMDB51 action classification, and 56.2% (+11.7% increase) Top-1 Recall on UCF101 nearest neighbor video retrieval.
翻译:对比性学习几乎缩小了受监管的图像显示学习与自我监督的图像显示学习之间的差距,也为视频进行了探索。然而,先前关于视频数据对比性学习的工作没有探讨明确鼓励在时间方面区分特征的效果。我们开发了一个新的时间对比性学习框架,包括两个新的损失,以改善现有的对比性自我监督的视频显示学习方法。当地时间对比性损失增加了对同一视频中非重叠片段的区分,而全球-地方时间对比性目标则是区分输入剪辑特征图的时步,以增加所学特征的时间多样性。我们拟议的时间对比性学习框架在各种下游视频理解任务中取得了显著的改进,如行动识别、有限标签行动分类和在多个视频数据集和主干线上最近的近邻视频检索。我们还展示了视觉相似类的微分级行动分类有显著改进。随着常用的3D ResNet-18结构,我们提出的时间对比性学习功能差异性学习框架(51%+51%)在前一至前一个最高级分类中实现了82.4%的递增幅度(51%的递增51%)和前一至前一个最接近的HCF级(51%的递增51%的递增),我们在最高分类上增加了51%的递增了51%的排名。