We present a new method to capture detailed human motion, sampling more than 1000 unique points on the body. Our method outputs highly accurate 4D (spatio-temporal) point coordinates and, crucially, automatically assigns a unique label to each of the points. The locations and unique labels of the points are inferred from individual 2D input images only, without relying on temporal tracking or any human body shape or skeletal kinematics models. Therefore, our captured point trajectories contain all of the details from the input images, including motion due to breathing, muscle contractions and flesh deformation, and are well suited to be used as training data to fit advanced models of the human body and its motion. The key idea behind our system is a new type of motion capture suit which contains a special pattern with checkerboard-like corners and two-letter codes. The images from our multi-camera system are processed by a sequence of neural networks which are trained to localize the corners and recognize the codes, while being robust to suit stretching and self-occlusions of the body. Our system relies only on standard RGB or monochrome sensors and fully passive lighting and the passive suit, making our method easy to replicate, deploy and use. Our experiments demonstrate highly accurate captures of a wide variety of human poses, including challenging motions such as yoga, gymnastics, or rolling on the ground.
翻译:我们提出一种新的方法来捕捉详细的人类运动,在身体上取样1000多个独特的点。我们的方法输出出高度精确的 4D(空间-时空)点坐标,并且关键地是,自动为每个点指定一个独特的标签。点的位置和独特的标签仅从单个的 2D 输入图像中推断,不依靠时间跟踪或任何人体形状或骨骼运动模型。因此,我们所捕获的点轨迹包含输入图像中的所有细节,包括呼吸、肌肉收缩和肉质变形等动作,非常适合用作培训数据,以适应人体及其运动的先进模型。我们系统的关键思想是新型的运动捕捉套装,它包含一种特殊的模式,带有像棋盘一样的角落和双字母代码。我们多摄像系统的图像是由神经网络序列处理的,这些神经网络经过培训,可以将角本地化并识别代码,同时能够适应体部的伸缩和自我隔离。我们的系统仅依靠标准 RGB 或单色模型的高级模型来应用, 并且完全的移动式的滚动式感动式传感器。