We propose a sim-to-real framework for dexterous manipulation which can generalize to new objects of the same category in the real world. The key of our framework is to train the manipulation policy with point cloud inputs and dexterous hands. We propose two new techniques to enable joint learning on multiple objects and sim-to-real generalization: (i) using imagined hand point clouds as augmented inputs; and (ii) designing novel contact-based rewards. We empirically evaluate our method using an Allegro Hand to grasp novel objects in both simulation and real world. To the best of our knowledge, this is the first policy learning-based framework that achieves such generalization results with dexterous hands. Our project page is available at https://yzqin.github.io/dexpoint
翻译:我们提议了一个模拟到现实的超脱操纵框架,这个框架可以推广到真实世界中同一类别的新物体。我们框架的关键是用点云输入和光滑的手来训练操纵政策。我们建议了两种新技术,以便能够在多个物体和光滑的简单化方面进行联合学习:(一) 利用想象中的手点云作为增强的投入;(二) 设计新的基于接触的奖励。我们用经验评估我们的方法,用阿勒格罗手在模拟和现实世界中捕捉新物品。我们最了解的是,这是第一个以政策学习为基础的框架,用光滑的手取得这种普遍化结果。我们的项目网页可在https://yzqin.github.io/dexpoint上查阅。