We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy to the real robot hand. We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. One key contribution of our system is that we construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. This provides an intuitive interface and avoid unstable human-robot hand retargeting for data collection, leading to large-scale and high quality data. Once the data is collected, the customized robot hand trajectories can be converted to different specified robot hands (models that are manufactured) to generate training demonstrations. With imitation learning using our data, we show large improvement over baselines with multiple complex manipulation tasks. Importantly, we show our learned policy is significantly more robust when transferring to the real robot. More videos can be found in the https://yzqin.github.io/dex-teleop-imitation .
翻译:我们建议用人类演示的多指机器人手进行模拟操作的模拟学习, 并将政策转移给真正的机器人手。 我们引入了新型的单相机远程操作系统, 以高效收集 3D 演示, 仅使用 iPad 和计算机。 我们系统的一个关键贡献是, 我们为每个用户在物理模拟器中安装一个定制的机器人手, 这个模拟器与操作器手的运动结构和形状相似。 这提供了直观界面, 并避免了数据采集的不稳定的人类机器人手重新定位, 从而导致大规模和高质量的数据。 一旦数据收集, 定制的机器人手轨迹可以转换为不同的指定机器人手( 正在制造的模型) 来生成培训演示。 我们通过使用我们的数据进行模拟学习, 显示对基准的大幅改进, 并且有多重复杂的操作任务 。 重要的是, 我们向真正的机器人传输时显示我们学到的政策非常有力。 更多的视频可以在 https://yzqin.gith.io/dex-teleopimatation中找到 。