As a beloved sport worldwide, dancing is getting integrated into traditional and virtual reality-based gaming platforms nowadays. It opens up new opportunities in the technology-mediated dancing space. These platforms primarily rely on passive and continuous human pose estimation as an input capture mechanism. Existing solutions are mainly based on RGB or RGB-Depth cameras for dance games. The former suffers in low-lighting conditions due to the motion blur and low sensitivity, while the latter is too power-hungry, has a low frame rate, and has limited working distance. With ultra-low latency, energy efficiency, and wide dynamic range characteristics, the event camera is a promising solution to overcome these shortcomings. We propose YeLan, an event camera-based 3-dimensional high-frequency human pose estimation(HPE) system that survives low-lighting conditions and dynamic backgrounds. We collected the world's first event camera dance dataset and developed a fully customizable motion-to-event physics-aware simulator. YeLan outperforms the baseline models in these challenging conditions and demonstrated robustness against different types of clothing, background motion, viewing angle, occlusion, and lighting fluctuations.
翻译:作为全世界敬爱的体育运动,舞蹈正逐渐融入传统和虚拟基于现实的游戏平台,在以技术为媒介的舞蹈空间开辟了新的机会。这些平台主要依靠被动和连续的人类形象估计作为输入捕捉机制。现有的解决方案主要基于 RGB 或 RGB-Deph 相机,用于舞蹈游戏。前者由于运动模糊和敏感度低而处于低亮状态,而后者则过于饥饿,框架率低,而且工作距离有限。由于超低的延缓度、能源效率和广泛的动态范围特点,活动相机是克服这些缺点的一个很有希望的解决办法。我们建议Yelan,一个基于事件摄像机的三维高频人类姿势估计(HPE)系统,在低光亮的条件和动态背景中生存下来。我们收集了世界上第一个活动相机舞蹈数据集,并开发了一个完全可定制的运动-活动物理觉模拟器。Yelan在这些具有挑战性的条件中超越了基线模型,并展示了抵御不同服装、背景运动、视野、视野、视野和照明的坚固度。