End-to-end Manipulator Calligraphy Planning via Variational Imitation Learning (End-to-end Manipulator Calligraphy Planning via Variational Imitation Learning)

Planning from demonstrations has shown promising results with the advances of deep neural networks. One of the most popular real-world applications is automated handwriting using a robotic manipulator. Classically it is simplified as a two-dimension problem. This representation is suitable for elementary drawings, but it is not sufficient for Japanese calligraphy or complex work of art where the orientation of a pen is part of the user expression. In this study, we focus on automated planning of Japanese calligraphy using a three-dimension representation of the trajectory as well as the rotation of the pen tip, and propose a novel deep imitation learning neural network that learns from expert demonstrations through a combination of images and pose data. The network consists of a combination of variational auto-encoder, bi-directional LSTM, and Multi-Layer Perceptron (MLP). Experiments are conducted in a progressive way, and results demonstrate that the proposed approach is successful in completion of tasks for real-world robots, overcoming the distribution shift problem in imitation learning. The source code and dataset will be public.

翻译：--- 通过变分模仿学习实现的末端执行器书法规划 Planning from demonstrations has shown promising results with the advances of deep neural networks. One of the most popular real-world applications is automated handwriting using a robotic manipulator. Classically it is simplified as a two-dimension problem. This representation is suitable for elementary drawings, but it is not sufficient for Japanese calligraphy or complex work of art where the orientation of a pen is part of the user expression. 在深度神经网络的发展下, 基于示教的规划已经取得了不错的结果. 其中一个最受欢迎的真实世界应用是机械臂自动书写. 传统上, 它被简化为一个二维问题. 这种表示法适用于基本的绘画, 但对于日本书法或有笔尖定向的复杂的艺术作品来说则不足够. In this study, we focus on automated planning of Japanese calligraphy using a three-dimension representation of the trajectory as well as the rotation of the pen tip, and propose a novel deep imitation learning neural network that learns from expert demonstrations through a combination of images and pose data. The network consists of a combination of variational auto-encoder, bi-directional LSTM, and Multi-Layer Perceptron (MLP). Experiments are conducted in a progressive way, and results demonstrate that the proposed approach is successful in completion of tasks for real-world robots, overcoming the distribution shift problem in imitation learning. The source code and dataset will be public. 本文侧重于用三维轨迹表示和笔尖旋转自动规划日本书法，并提出了一种新的深度模仿学习神经网络，通过图像和姿态数据的组合从专家的示范中学习。这个网络由变分自编码器，双向LSTM和多层感知器(MLP)的组合组成。实验是按照一种逐步的方式进行的，结果表明，所提出的方法成功地完成了真实世界机器人的任务，克服了模仿学习中的分布转移问题。源代码和数据集将会公开。