High-quality 4D reconstruction of human performance with complex interactions to various objects is essential in real-world scenarios, which enables numerous immersive VR/AR applications. However, recent advances still fail to provide reliable performance reconstruction, suffering from challenging interaction patterns and severe occlusions, especially for the monocular setting. To fill this gap, in this paper, we propose RobustFusion, a robust volumetric performance reconstruction system for human-object interaction scenarios using only a single RGBD sensor, which combines various data-driven visual and interaction cues to handle the complex interaction patterns and severe occlusions. We propose a semantic-aware scene decoupling scheme to model the occlusions explicitly, with a segmentation refinement and robust object tracking to prevent disentanglement uncertainty and maintain temporal consistency. We further introduce a robust performance capture scheme with the aid of various data-driven cues, which not only enables re-initialization ability, but also models the complex human-object interaction patterns in a data-driven manner. To this end, we introduce a spatial relation prior to prevent implausible intersections, as well as data-driven interaction cues to maintain natural motions, especially for those regions under severe human-object occlusions. We also adopt an adaptive fusion scheme for temporally coherent human-object reconstruction with occlusion analysis and human parsing cue. Extensive experiments demonstrate the effectiveness of our approach to achieve high-quality 4D human performance reconstruction under complex human-object interactions whilst still maintaining the lightweight monocular setting.
翻译:为了填补这一空白,我们在本文件中提议采用强力四维系统,为人类物体互动情景建立一个强大的量性性性能重建系统,仅使用单一的 RGBD 传感器,该传感器将各种数据驱动的视觉和互动信号信号结合起来,以便处理复杂的互动模式和严重隐蔽性。我们提议采用一种具有超小型的复杂复杂场景分解计划,以明确模拟隐蔽性重建,同时进行分解性完善和严格的对象跟踪,以防止分解性不确定性并保持时间一致性。我们进一步提出一种强大的性能采集计划,借助各种数据驱动的提示,不仅能够重新定位能力,而且以数据驱动的方式模拟复杂的人类弹道互动模式。我们为此提出一种空间关系,以预防复杂的互动模式复杂互动模式和严重隐蔽性隐蔽性隐蔽性。我们提议在人类隐蔽性场景进行明确的隐蔽性脱钩,同时进行分解,并进行严格的物体分解性目标跟踪,以防止引起分解的不确定性的不确定性,并保持时间的一致性。我们进一步引入一个强有力的性性性性性捕捉摸测系统,在数据驱动下,同时进行人类的人类持续地进行人类感性循环的精确的模拟。