Haptic-enabled Mixed Reality System for Mixed-initiative Remote Robot Control

Robots assist in many areas that are considered unsafe for humans to operate. For instance, in handling pandemic diseases such as the recent Covid-19 outbreak and other outbreaks like Ebola, robots can assist in reaching areas dangerous for humans and do simple tasks such as pick up the correct medicine (among a set of bottles prescribed) and deliver to patients. In such cases, it might not be good to rely on the fully autonomous operation of robots. Since many mobile robots are fully functional with low-level tasks such as grabbing and moving, we consider the mixed-initiative control where the user can guide the robot remotely to finish such tasks. For this mixed-initiative control, the user controlling the robot needs to visualize a 3D scene as seen by the robot and guide it. Mixed reality can virtualize reality and immerse users in the 3D scene that is reconstructed from the real-world environment. This technique provides the user more freedom such as choosing viewpoints at view time. In recent years, benefiting from the high-quality data from Light Detection and Ranging (LIDAR) and RGBD cameras, mixed reality is widely used to build networked platforms to improve the performance of robot teleoperations and robot-human collaboration, and enhanced feedback for mixed-initiative control. In this paper, we proposed a novel haptic-enabled mixed reality system, that provides haptic interfaces to interact with the virtualized environments and give remote guidance for mobile robots towards high-level tasks. The experimental results show the effectiveness and flexibility of the proposed haptic enabled mixed reality system.