We outline the role of forward and inverse modelling approaches in the design of human--computer interaction systems. Causal, forward models tend to be easier to specify and simulate, but HCI requires solutions of the inverse problem. We infer finger 3D position $(x,y,z)$ and pose (pitch and yaw) on a mobile device using capacitive sensors which can sense the finger up to 5cm above the screen. We use machine learning to develop data-driven models to infer position, pose and sensor readings, based on training data from: 1. data generated by robots, 2. data from electrostatic simulators 3. human-generated data. Machine learned emulation is used to accelerate the electrostatic simulation performance by a factor of millions. We combine a Conditional Variational Autoencoder with domain expertise/models experimentally collected data. We compare forward and inverse model approaches to direct inference of finger pose. The combination gives the most accurate reported results on inferring 3D position and pose with a capacitive sensor on a mobile device.
翻译:我们概述了前方和反向建模方法在设计人-计算机互动系统中的作用。 由前方模型往往更容易指定和模拟,但HCI需要反向问题的解决方案。 我们用能感应到手指到屏幕上5厘米的感应器,在移动设备上推断3D方位(x,y,z),并显示(pitch和yaw)。 我们利用机器学习开发数据驱动模型,根据以下培训数据推导位置、显示和感应读数:1. 机器人生成的数据,2. 电子模拟器生成的数据,3. 人类生成的数据。 机器学习的模拟用百万倍数加速电磁模拟性性能。 我们将致电自动电解器与实验收集的域专门知识/模型结合起来。 我们用前方和反向模型方法比较手指姿势的推断。 组合提供了最准确的预测3D位置和与移动设备上电动感应传感器的结果。