GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Speech Emotion Recognition

Contrastive learning based cross-modality pretraining approaches have recently exhibited impressive success in diverse fields. In this paper, we propose GEmo-CLAP, a kind of gender-attribute-enhanced contrastive language-audio pretraining (CLAP) method for speech emotion recognition. Specifically, a novel emotion CLAP model (Emo-CLAP) is first built, utilizing pre-trained WavLM and RoBERTa models. Second, given the significance of the gender attribute in speech emotion modeling, two novel soft label based GEmo-CLAP (SL-GEmo-CLAP) and multi-task learning based GEmo-CLAP (ML-GEmo-CLAP) models are further proposed to integrate emotion and gender information of speech signals, forming more reasonable objectives. Extensive experiments on IEMOCAP show that our proposed two GEmo-CLAP models consistently outperform the baseline Emo-CLAP, while also achieving the best recognition performance compared with recent state-of-the-art methods. Noticeably, the proposed SL-GEmo-CLAP model achieves the best UAR of 81.43\% and WAR of 83.16\% which performs better than other state-of-the-art SER methods by at least 3\%.