MET: Multimodal Perception of Engagement for Telehealth

We present MET, a learning-based algorithm for perceiving a human's level of engagement from videos that give us access to only the face, speech and text. We leverage latent vectors corresponding to Affective and Cognitive features frequently used in psychology literature to understand a person's level of engagement in a semi-supervised GAN-based framework. The method is extremely useful in the case of telehealth. We showcase the efficacy of this method from the perspective of mental health and more specifically how this can be leveraged for a better understanding of patient engagement during telemental health sessions. We also explore the usefulness of our framework and contrast it against existing works in being able to estimate another important mental health indicator, namely valence, and arousal. Our framework reports 40% improvements in RMSE over SOTA method in Engagement Regression and 50% improvements in RMSE over SOTA method in Valence-Arousal Regression. To tackle the scarcity of publicly available datasets in the telemental health space, we release a new dataset, MEDICA, for mental health patient engagement detection. Our dataset, MEDICA consists of 1299 videos, each 3 seconds long. To the best of our knowledge, our approach is the first method capable to model telemental health session data based on psychology-driven Affective and Cognitive features, which also accounts for data sparsity by leveraging a semi-supervised setup. To assert the usefulness of our method, we will also compare the association of the engagement values obtained from our model with the other engagement measures used by psychotherapists.