Recognizing facial activity is a well-understood (but non-trivial) computer vision problem. However, reliable solutions require a camera with a good view of the face, which is often unavailable in wearable settings. Furthermore, in wearable applications, where systems accompany users throughout their daily activities, a permanently running camera can be problematic for privacy (and legal) reasons. This work presents an alternative solution based on the fusion of wearable inertial sensors, planar pressure sensors, and acoustic mechanomyography (muscle sounds). The sensors were placed unobtrusively in a sports cap to monitor facial muscle activities related to facial expressions. We present our integrated wearable sensor system, describe data fusion and analysis methods, and evaluate the system in an experiment with thirteen subjects from different cultural backgrounds (eight countries) and both sexes (six women and seven men). In a one-model-per-user scheme and using a late fusion approach, the system yielded an average F1 score of 85.00% for the case where all sensing modalities are combined. With a cross-user validation and a one-model-for-all-user scheme, an F1 score of 79.00% was obtained for thirteen participants (six females and seven males). Moreover, in a hybrid fusion (cross-user) approach and six classes, an average F1 score of 82.00% was obtained for eight users. The results are competitive with state-of-the-art non-camera-based solutions for a cross-user study. In addition, our unique set of participants demonstrates the inclusiveness and generalizability of the approach.
翻译:承认面部活动是一个非常清楚(但非三重)的计算机视觉问题。然而,可靠的解决方案需要一台清晰可见的照相机,其面孔往往在可磨损的环境下得不到。此外,在可磨损的应用程序中,系统陪伴用户在其日常活动中,一个永久运行的照相机可能会因隐私(和法律)原因产生问题。这项工作提供了一个基于可磨损惯性感应器、平压感应器和声学混合法(肌肉声音)的替代解决方案。传感器被安放在一个体育帽中,以监测与面部表达有关的面部肌肉活动。我们展示了我们的综合可磨损感应传感器系统,描述数据聚合和分析方法,并在对来自不同文化背景(8个国家)和男女(6个女性和7个男性)的13个实验中评估了该系统。在1个模型-每个用户计划中,在所有感测方法组合的情况下,系统平均得出85.00 %的F1分。在交叉用户验证和1个模型上与8个非用户计划之间,1个用户的F1级标准是8个比例。在8个用户中,1级和13个用户的平均比例为1个用户。