We present a differentiable pipeline for simulating the motion of objects that represent their geometry as a continuous density field parameterized as a deep network. This includes Neural Radiance Fields (NeRFs), and other related models. From the density field, we estimate the dynamical properties of the object, including its mass, center of mass, and inertia matrix. We then introduce a differentiable contact model based on the density field for computing normal and friction forces resulting from collisions. This allows a robot to autonomously build object models that are visually and dynamically accurate from still images and videos of objects in motion. The resulting Dynamics-Augmented Neural Objects (DANOs) are simulated with an existing differentiable simulation engine, Dojo, interacting with other standard simulation objects, such as spheres, planes, and robots specified as URDFs. A robot can use this simulation to optimize grasps and manipulation trajectories of neural objects, or to improve the neural object models through gradient-based real-to-simulation transfer. We demonstrate the pipeline to learn the coefficient of friction of a bar of soap from a real video of the soap sliding on a table. We also learn the coefficient of friction and mass of a Stanford bunny through interactions with a Panda robot arm from synthetic data, and we optimize trajectories in simulation for the Panda arm to push the bunny to a goal location.
翻译:我们为模拟以连续密度场为深度网络的连续密度场参数来代表其几何的物体运动展示一个不同的管道。 其中包括神经辐射场( NERFs) 和其他相关模型。 我们从密度场中估计物体的动态特性, 包括质量、 质量中心、 惯性矩阵等。 然后我们推出一个基于密度域的不同接触模型, 用于计算碰撞产生的正常力和摩擦力。 这允许机器人自主地构建从静止物体的图像和视频中直观和动态地准确的物体模型。 由此产生的动态增强神经物体( DaNOs) 模拟了现有的不同模拟引擎。 Dojo, 与其他标准模拟物体( 如表面、 质量中心、 质量矩阵和惯性矩阵矩阵 ) 进行互动。 一个机器人可以使用这种模拟来优化抓取和操纵神经物体的轨迹轨迹, 或者通过基于梯度的实际图像转换来改进神经物体模型模型模型模型。 我们展示了管道, 学习了以可变动的模型的摩擦度系数, 以及从一个真实的模型的模型, 我们从一个模型的模型的模型的模型的模型的模型的模型的模型, 以及一个模型的模型的模型的模型的模型的模型的模型的模型的模型的模型的模型的模型的模型, 和模型的模型的模型的模型的模型的模型的模型的模型。