Sim-to-real transfer is a powerful paradigm for robotic reinforcement learning. The ability to train policies in simulation enables safe exploration and large-scale data collection quickly at low cost. However, prior works in sim-to-real transfer of robotic policies typically do not involve any human-robot interaction because accurately simulating human behavior is an open problem. In this work, our goal is to leverage the power of simulation to train robotic policies that are proficient at interacting with humans upon deployment. But there is a chicken and egg problem -- how to gather examples of a human interacting with a physical robot so as to model human behavior in simulation without already having a robot that is able to interact with a human? Our proposed method, Iterative-Sim-to-Real (i-S2R), attempts to address this. i-S2R bootstraps from a simple model of human behavior and alternates between training in simulation and deploying in the real world. In each iteration, both the human behavior model and the policy are refined. For all training we apply a new evolutionary search algorithm called Blackbox Gradient Sensing (BGS). We evaluate our method on a real world robotic table tennis setting, where the objective for the robot is to play cooperatively with a human player for as long as possible. Table tennis is a high-speed, dynamic task that requires the two players to react quickly to each other's moves, making for a challenging test bed for research on human-robot interaction. We present results on an industrial robotic arm that is able to cooperatively play table tennis with human players, achieving rallies of 22 successive hits on average and 150 at best. Further, for 80% of players, rally lengths are 70% to 175% longer compared to the sim-to-real plus fine-tuning (S2R+FT) baseline. For videos of our system in action, please see https://sites.google.com/view/is2r.
翻译:超到真实的传输是机器人强化学习的强大范例。 模拟中训练政策的能力使得能够安全地探索和快速低成本地收集大规模数据。 然而, 机器人政策的模拟到真实的传输通常不涉及任何人类机器人互动, 因为准确模拟人类行为是一个开放的问题。 在这项工作中, 我们的目标是利用模拟的力量来训练在部署时能够与人类互动的机器人政策。 但是, 存在一个鸡蛋问题 -- 如何收集人类与物理机器人互动的例子, 从而在模拟中模拟人类行为, 而不是已经拥有一个能够与人类互动的机器人? 但是, 我们提出的机器人的模拟到真实的机器人政策通常不会涉及任何人类机器人互动, 因为准确模拟人类行为是一个开放的问题。 i- S2R 的靴系从一个简单的人类行为模型到模拟和在现实世界中部署的训练。 每一次的模拟、 人类行为模型到政策都得到了精细精细的。 在所有培训中, 我们用新的进化搜索算算算算法, 叫做黑盒的精确度测测算( BGS) (BY), 我们用一个快速的机器人动作在每部的游戏中, 一个高速度, 我们用一个人类的机器人操作的机器人操作的游戏是一个高速度, 一个人类的游戏, 一个人类的游戏, 需要一个人类的游戏, 一个真正的游戏的游戏, 一个真正的游戏, 一个高速度, 一个真正的机器人的游戏, 一个人类的游戏, 一个人类的游戏, 一个真正的游戏, 一个人类的游戏, 一个高速度, 一个真正的游戏, 一个人类的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的机器人, 一个在游戏, 一个真正的游戏, 一个真正的游戏, 一个在游戏, 一个真正的游戏, 一个真正的游戏, 一个上, 一个上, 一个人类的游戏, 一个人类的游戏, 一个人类的游戏, 一个稳定的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个真正的游戏, 一个上, 一个上, 一个具有一个游戏, 一个稳定的游戏, 一个人类的游戏, 一个稳定的游戏, 一个稳定的游戏, 一个稳定的游戏, 一个稳定的游戏, 一个上, 一个高的游戏,