Spatial convolutions are widely used in numerous deep video models. It fundamentally assumes spatio-temporal invariance, i.e., using shared weights for every location in different frames. This work presents Temporally-Adaptive Convolutions (TAdaConv) for video understanding, which shows that adaptive weight calibration along the temporal dimension is an efficient way to facilitate modelling complex temporal dynamics in videos. Specifically, TAdaConv empowers the spatial convolutions with temporal modelling abilities by calibrating the convolution weights for each frame according to its local and global temporal context. Compared to previous temporal modelling operations, TAdaConv is more efficient as it operates over the convolution kernels instead of the features, whose dimension is an order of magnitude smaller than the spatial resolutions. Further, the kernel calibration also brings an increased model capacity. We construct TAda2D networks by replacing the spatial convolutions in ResNet with TAdaConv, which leads to on par or better performance compared to state-of-the-art approaches on multiple video action recognition and localization benchmarks. We also demonstrate that as a readily plug-in operation with negligible computation overhead, TAdaConv can effectively improve many existing video models with a convincing margin. Codes and models will be made available at https://github.com/alibaba-mmai-research/pytorch-video-understanding.
翻译:在许多深层视频模型中广泛使用空间空间共变。 它基本上假定时空差异, 即使用不同框架中每个位置的共享权重。 这项工作展示了用于视频理解的“ 时间- 适应性进化” (Tada Conv) 视频理解, 表明时间维度的适应性权重校准是便利模拟视频中复杂时间动态的有效方法。 具体地说, TAda Conv 通过根据每个框架的本地和全球时间背景校准其时间建模能力, 赋予空间共变能力以时间性变异能力。 与以往的时间建模操作相比, TAda Conv 效率更高, 因为它在变动内核内核运行, 而不是其尺寸小于空间分辨率。 此外, 内核校校准还带来一个更大的模型能力。 我们建造TAda2D网络, 将ResNet的空间共变换成TAda Convonvon, 与多个视频动作识别和本地化基准的状态方法相比, 将提高性或更好的性性表现。 我们还展示了可轻易得到的图像/ 模型。