Robots assisting us in factories or homes must learn to make use of objects as tools to perform tasks, e.g., a tray for carrying objects. We consider the problem of learning commonsense knowledge of when a tool may be useful and how its use may be composed with other tools to accomplish a high-level task instructed by a human. We introduce TANGO, a novel neural model for predicting task-specific tool interactions. TANGO is trained using demonstrations obtained from human teachers instructing a virtual robot in a physics simulator. TANGO encodes the world state comprising of objects and symbolic relationships between them using a graph neural network. The model learns to attend over the scene using knowledge of the goal and the action history, finally decoding the symbolic action to execute. Crucially, we address generalization to unseen environments where some known tools are missing, but alternative unseen tools are present. We show that by augmenting the representation of the environment with pre-trained embeddings derived from a knowledge-base, the model can generalize effectively to novel environments. Experimental results show a 60.5-78.9% improvement over the baseline in predicting successful symbolic plans in unseen settings for a simulated mobile manipulator.
翻译:协助我们在工厂或家中工作的机器人必须学会将物体用作执行任务的工具,例如携带物体的托盘。我们考虑的问题是,学习关于工具何时有用以及如何将其使用与其他工具相结合的常识知识,以完成由人类指示的高级任务。我们介绍TANGO,这是一个用于预测任务特定工具互动的新型神经模型。TANGO是利用人类教师在物理学模拟器中指导虚拟机器人的演示来接受培训的。TANGO用图形神经网络来编码由物体和它们之间的象征关系组成的世界状态。模型学会利用对目标和行动历史的知识在现场上关注,最终解码要执行的象征性行动。我们非常明确地将一些已知工具缺失的无形环境概括化,但也有其他的不可见工具。我们显示,通过从知识库中获得的经过预先训练的嵌入器来增加环境的代表性,该模型可以有效地向新环境普及。实验结果显示,在预测成功的模拟模型的模型中,60.5-78.9%的移动模型在模拟模型中显示,60.5-78.9%的移动模型将改进到一个成功的模拟模型。