A Novel Transfer Learning Framework with Prototypical Representation based Pairwise Learning for Cross-Subject Cross-Session EEG-Based Emotion Recognition

In this paper, we propose a novel transfer learning framework with Prototypical Representation based Pairwise Learning (PR-PL) for EEG-based emotion recognition, which learns discriminative and generalized prototypical representations for emotion revealing across individuals and formulates emotion recognition as pairwise learning to alleviate the model reliance on precise label information. Prototypical learning based adversarial discriminative domain adaptation method is developed to encode the inherent emotion-related semantic structure of EEG data. Pairwise learning with an adaptive pseudo-labeling method is proposed to achieve a reliable and stable model learning with noisy labels. Through domain adaptation, besides aligning the feature representation of the source and the target on a shared feature space, the feature separability of both source and target domains is also considered. The characterized prototypical representations are evident with a high feature concentration within one single emotion category and a high feature separability across different emotion categories. Extensive experiments are conducted on two benchmark databases (SEED and SEED-IV) for recognizing three and four emotion categories using four cross-validation evaluation protocols. The experimental results demonstrate the superiority of the proposed PR-PL against the state-of-the-art methods on cross-subject cross-session, cross-subject within-session, within-subject cross-session, and within-subject within-session evaluation protocols, which shows the power of PR-PL in dealing with the ambiguity of neurophysiological responses in affective studies.