Self-supervised skeleton-based action recognition with contrastive learning has attracted much attention. Recent literature shows that data augmentation and large sets of contrastive pairs are crucial in learning such representations. In this paper, we found that directly extending contrastive pairs based on normal augmentations brings limited returns in terms of performance, because the contribution of contrastive pairs from the normal data augmentation to the loss get smaller as training progresses. Therefore, we delve into hard contrastive pairs for contrastive learning. Motivated by the success of mixing augmentation strategy which improves the performance of many tasks by synthesizing novel samples, we propose SkeleMixCLR: a contrastive learning framework with a spatio-temporal skeleton mixing augmentation (SkeleMix) to complement current contrastive learning approaches by providing hard contrastive samples. First, SkeleMix utilizes the topological information of skeleton data to mix two skeleton sequences by randomly combing the cropped skeleton fragments (the trimmed view) with the remaining skeleton sequences (the truncated view). Second, a spatio-temporal mask pooling is applied to separate these two views at the feature level. Third, we extend contrastive pairs with these two views. SkeleMixCLR leverages the trimmed and truncated views to provide abundant hard contrastive pairs since they involve some context information from each other due to the graph convolution operations, which allows the model to learn better motion representations for action recognition. Extensive experiments on NTU-RGB+D, NTU120-RGB+D, and PKU-MMD datasets show that SkeleMixCLR achieves state-of-the-art performance. Codes are available at https://github.com/czhaneva/SkeleMixCLR.
翻译:自我监督的基于骨架的动作识别与对比学习已经引起很多关注 。 最近的文献显示, 数据增强和大量对比配对对于学习这些表达方式至关重要 。 在本文中, 我们发现, 直接扩展基于正常增强( SkeleMix) 的对比配对可以带来有限的性能回报, 因为正常数据增强的对比配对对于损失的贡献随着培训进展而变得更小 。 因此, 我们通过随机梳理裁剪裁剪的骨架碎片( 剪剪裁的视图), 使许多任务的性能通过合成新样本而得到改善 。 我们建议 SkeleMix: 一个带有 spatio- 脉冲骨架放大( SkeleMix) 混合的对比配对式对配对配对, 通过提供硬对比样本来补充当前的对比学习方法 。 SkeleamMix 利用骨架数据的表信息将两个骨架序列混在一起, 裁剪裁的骨架碎片( 缩视图) 和其余的骨架序列( 调视图) 。 第二, spate- mix- mix- mix mode- moud- mode- massal mode- mode- lixal 将两次浏览显示这些直径 。