Existing learning approaches to dexterous manipulation use demonstrations or interactions with the environment to train black-box neural networks that provide little control over how the robot learns the skills or how it would perform post training. These approaches pose significant challenges when implemented on physical platforms given that, during initial stages of training, the robot's behavior could be erratic and potentially harmful to its own hardware, the environment, or any humans in the vicinity. A potential way to address these limitations is to add constraints during learning that restrict and guide the robot's behavior during training as well as roll outs. Inspired by the success of constrained approaches in other domains, we investigate the effects of adding position-based constraints to a 24-DOF robot hand learning to perform object relocation using Constrained Policy Optimization. We find that a simple geometric constraint can ensure the robot learns to move towards the object sooner than without constraints. Further, training with this constraint requires a similar number of samples as its unconstrained counterpart to master the skill. These findings shed light on how simple constraints can help robots achieve sensible and safe behavior quickly and ease concerns surrounding hardware deployment. We also investigate the effects of the strictness of these constraints and report findings that provide insights into how different degrees of strictness affect learning outcomes. Our code is available at https://github.com/GT-STAR-Lab/constrained-rl-dexterous-manipulation.
翻译:极速操纵的现有学习方法使用演示或与环境互动来训练黑盒神经网络,这些网络对机器人如何学习技能或如何进行事后培训几乎没有什么控制。这些方法在实际平台上实施时构成重大挑战,因为在培训的初始阶段,机器人的行为可能变化不定,并可能对其自身硬件、环境或附近任何人造成潜在伤害。解决这些限制的一个潜在办法是,在学习过程中增加限制和指导机器人在培训和推出过程中的行为的制约。在其他领域的受限方法取得成功的启发下,我们调查将基于位置的限制添加到24DOF机器人手学习如何使用 Consstrated Policy进行物体迁移的影响。我们发现,简单的几何等限制可以确保机器人早于不受限制地向目标移动。此外,关于这种限制的培训需要与掌握技能的不受限制的对应方类似数量的样本。这些发现揭示了如何简单的限制可以帮助机器人迅速实现明智和安全的行为,并减轻了对硬件部署的关切。我们还研究了如何严格地了解这些限制的结果。