2D+3D facial expression recognition (FER) can effectively cope with illumination changes and pose variations by simultaneously merging 2D texture and more robust 3D depth information. Most deep learning-based approaches employ the simple fusion strategy that concatenates the multimodal features directly after fully-connected layers, without considering the different degrees of significance for each modality. Meanwhile, how to focus on both 2D and 3D local features in salient regions is still a great challenge. In this letter, we propose the adaptive fusion network with masks (AFNet-M) for 2D+3D FER. To enhance 2D and 3D local features, we take the masks annotating salient regions of the face as prior knowledge and design the mask attention module (MA) which can automatically learn two modulation vectors to adjust the feature maps. Moreover, we introduce a novel fusion strategy that can perform adaptive fusion at convolutional layers through the designed importance weights computing module (IWC). Experimental results demonstrate that our AFNet-M achieves the state-of-the-art performance on BU-3DFE and Bosphorus datasets and requires fewer parameters in comparison with other models.
翻译:2D+3D 面部表情识别(FER) 能够有效应对光化变化,并通过同时合并 2D 纹理和更强健的 3D 深度信息而带来变异。 大多数深层次的学习方法都采用简单的聚合战略,将多式联运特征直接结合在完全连接的层之后,而没有考虑每种模式的不同意义程度。与此同时,如何关注突出区域的 2D 和 3D 地方特征仍然是一个巨大的挑战。在本信中,我们建议为 2D+3D FER 建立带有2D 顶罩(AFNet-M) 的适应性融合网络。为了加强 2D 和 3D 本地特征,我们把标记突出面部特征的面部为先前的知识,并设计面具关注模块(MA),该模块可以自动学习两个调制导矢量来调整地图。此外,我们引入了一种新的融合战略,可以通过设计的重要重量计算模块(IWC) 来在革命层进行适应性融合。实验结果表明,我们的AFNet-M 实现B-3DFE 和Bosphorus 数据设置的其他参数比较要求较少。