Natural language is one of the most intuitive ways to express human intent. However, translating instructions and commands towards robotic motion generation and deployment in the real world is far from being an easy task. The challenge of combining a robot's inherent low-level geometric and kinodynamic constraints with a human's high-level semantic instructions traditionally is solved using task-specific solutions with little generalizability between hardware platforms, often with the use of static sets of target actions and commands. This work instead proposes a flexible language-based framework that allows a user to modify generic robotic trajectories. Our method leverages pre-trained language models (BERT and CLIP) to encode the user's intent and target objects directly from a free-form text input and scene images, fuses geometrical features generated by a transformer encoder network, and finally outputs trajectories using a transformer decoder, without the need of priors related to the task or robot information. We significantly extend our own previous work presented in Bucker et al. by expanding the trajectory parametrization space to 3D and velocity as opposed to just XY movements. In addition, we now train the model to use actual images of the objects in the scene for context (as opposed to textual descriptions), and we evaluate the system in a diverse set of scenarios beyond manipulation, such as aerial and legged robots. Our simulated and real-life experiments demonstrate that our transformer model can successfully follow human intent, modifying the shape and speed of trajectories within multiple environments. Codebase available at: https://github.com/arthurfenderbucker/LaTTe-Language-Trajectory-TransformEr.git
翻译:自然语言是表达人类意图的最直观的方法之一。 然而, 将指令和指令翻译为机器人运动生成和在现实世界部署远非易事。 将机器人固有的低水平几何和运动动力限制与人类高层次语义指令相结合的挑战, 通常通过使用固定目标动作和命令的固定组合, 而在硬件平台之间往往使用固定的目标动作和命令, 使用任务特定解决方案, 而很少具有一般性。 这项工作提议一个灵活的语言框架, 让用户能够修改通用机器人轨迹。 我们的方法将预训练的语言模型( ERT 和 CLIP) 用于将用户的意向和目标直接从自由格式的文本输入和场景图像中编码。 由变异器编码网络生成的几何数学特征结合, 使用变异器解调器, 不需要与任务或机器人信息相关的先前版本。 我们大大扩展了自己在巴克尔和阿尔等人体内展示的可使用的工作。 此外, 我们把轨迹折变变变频空间到轨迹模型, 现在和速度的轨迹变动中, 我们用实时的轨迹变变变变变变图,, 直为X 的变变变变变变变变的变的轨道系统, 系统, 我们的变变变变变的变的变变变变的变的变变变变的变的变的变的变的变变变的变的变的变的变的变的变变的变的变的变的变的变的变的变的变的变的变的变的变的系统, 。