A novel concept of vision-based intelligent control of robotic arms is developed here in this work. This work enables the controlling of robotic arms motion only with visual inputs, that is, controlling by showing the videos of correct movements. This work can broadly be sub-divided into two segments. The first part of this work is to develop an unsupervised vision-based method to control robotic arm in 2-D plane, and the second one is with deep CNN in the same task in 3-D plane. The first method is unsupervised, where our aim is to perform mimicking of human arm motion in real-time by a manipulator. We developed a network, namely the vision-to-motion optical network (DON), where the input should be a video stream containing hand movements of human, the the output would be out the velocity and torque information of the hand movements shown in the videos. The output information of the DON is then fed to the robotic arm by enabling it to generate motion according to the real hand videos. The method has been tested with both live-stream video feed as well as on recorded video obtained from a monocular camera even by intelligently predicting the trajectory of human hand hand when it gets occluded. This is why the mimicry of the arm incorporates some intelligence to it and becomes intelligent mimic (i-mimic). Alongside the unsupervised method another method has also been developed deploying the deep neural network technique with CNN (Convolutional Neural Network) to perform the mimicking, where labelled datasets are used for training. The same dataset, as used in the unsupervised DON-based method, is used in the deep CNN method, after manual annotations. Both the proposed methods are validated with off-line as well as with on-line video datasets in real-time. The entire methodology is validated with real-time 1-link and simulated n-link manipulators alongwith suitable comparisons.
翻译:在这项工作中开发了一个基于视觉的机器人武器智能控制的新概念。 这项工作只能通过视觉输入来控制机器人武器运动, 也就是通过显示正确运动的视频来控制机器人武器运动。 这项工作可以大致分为两个部分。 这项工作的第一部分是开发一种不受监督的基于视觉的方法来控制2D平面的机器人手臂, 其次是在3D平面的同一任务中与深CNN一起。 第一个方法不受监督, 我们的目标是通过一个操纵者实时进行模拟人体手臂运动。 我们开发了一个网络, 即视觉到感动的光学网络( DON), 这个网络的输入应该是包含人类手动运动的视频流, 输出将超出视频中显示的手动动作的速度和感光度信息。 然后, DON的输出信息被输入到机械臂上, 使得它能够根据真正的手动视频产生运动动作。 这个方法已经用实时的视频进行测试, 以及从右心动摄像机获得的非录影带的视频, 即使是在智能的网络中进行深度的, 也通过智能的精确的机路路路路段, 将数据转换到另一种手动方法。 。 将使用另一个的机路路路路路路路路,,, 将使用, 将使用, 将使用, 将使用, 手法, 手法将使用另一个的方法是手法, 手法, 手法, 机路机路路路路路路路路路路路路路路路,,, 。