XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning

We present XKD, a novel self-supervised framework to learn meaningful representations from unlabelled video clips. XKD is trained with two pseudo tasks. First, masked data reconstruction is performed to learn individual representations from audio and visual streams. Next, self-supervised cross-modal knowledge distillation is performed between the two modalities through teacher-student setups to learn complementary information. To identify the most effective information to transfer and also to tackle the domain gap between audio and visual modalities which could hinder knowledge transfer, we introduce a domain alignment and feature refinement strategy for effective cross-modal knowledge distillation. Lastly, to develop a general-purpose network capable of handling both audio and visual streams, modality-agnostic variants of our proposed framework are introduced, which use the same backbone for both audio and visual modalities. Our proposed cross-modal knowledge distillation improves linear evaluation top-1 accuracy of video action classification by 8.6% on UCF101, 8.2% on HMDB51, 13.9% on Kinetics-Sound, and 15.7% on Kinetics400. Additionally, our modality-agnostic variant shows promising results in developing a general-purpose network capable of learning both data streams for solving different downstream tasks.