We propose a deep videorealistic 3D human character model displaying highly realistic shape, motion, and dynamic appearance learned in a new weakly supervised way from multi-view imagery. In contrast to previous work, our controllable 3D character displays dynamics, e.g., the swing of the skirt, dependent on skeletal body motion in an efficient data-driven way, without requiring complex physics simulation. Our character model also features a learned dynamic texture model that accounts for photo-realistic motion-dependent appearance details, as well as view-dependent lighting effects. During training, we do not need to resort to difficult dynamic 3D capture of the human; instead we can train our model entirely from multi-view video in a weakly supervised manner. To this end, we propose a parametric and differentiable character representation which allows us to model coarse and fine dynamic deformations, e.g., garment wrinkles, as explicit space-time coherent mesh geometry that is augmented with high-quality dynamic textures dependent on motion and view point. As input to the model, only an arbitrary 3D skeleton motion is required, making it directly compatible with the established 3D animation pipeline. We use a novel graph convolutional network architecture to enable motion-dependent deformation learning of body and clothing, including dynamics, and a neural generative dynamic texture model creates corresponding dynamic texture maps. We show that by merely providing new skeletal motions, our model creates motion-dependent surface deformations, physically plausible dynamic clothing deformations, as well as video-realistic surface textures at a much higher level of detail than previous state of the art approaches, and even in real-time.
翻译:我们提出一个深度视频现实的 3D 人性模型, 展示高度现实的形状、 运动和动态外观, 展示出由多视图图像以新的微弱监管方式学习的外形、 运动和动态外观。 与以往的工作不同, 我们的可控的 3D 字符显示动态, 例如, 裙子的摆动, 依靠骨骼身体运动, 不需要复杂的物理模拟。 我们的性格模型还包含一个深层次的动态纹理模型, 描述光- 现实运动的外观细节, 以及依赖视觉的照明效应。 在培训过程中, 我们不需要用困难的动态3D 捕捉人类的动态3D ; 相反, 我们可以用微弱监管的方式将我们的模型完全从多视角视频视频视频视频中培养出来。 为此, 我们提议了一个准度的和不同的性格表示, 使我们能够模拟粗和细的动态变形变形结构, 我们用新的变形结构来构建一个动态的变形结构, 我们用新的变形的变形结构, 我们用新的变形的变形的变形结构, 以不断的变形的变形的变形的变形的变形的变形结构, 以不断变形的变形的变形的变形的变形的变形的变形的变形的变形的变形的变形的变形图。