In this article, we propose a novel LiDAR and event camera fusion modality for subterranean (SubT) environments for fast and precise object and human detection in a wide variety of adverse lighting conditions, such as low or no light, high-contrast zones and in the presence of blinding light sources. In the proposed approach, information from the event camera and LiDAR are fused to localize a human or an object-of-interest in a robot's local frame. The local detection is then transformed into the inertial frame and used to set references for a Nonlinear Model Predictive Controller (NMPC) for reactive tracking of humans or objects in SubT environments. The proposed novel fusion uses intensity filtering and K-means clustering on the LiDAR point cloud and frequency filtering and connectivity clustering on the events induced in an event camera by the returning LiDAR beams. The centroids of the clusters in the event camera and LiDAR streams are then paired to localize reflective markers present on safety vests and signs in SubT environments. The efficacy of the proposed scheme has been experimentally validated in a real SubT environment (a mine) with a Pioneer 3AT mobile robot. The experimental results show real-time performance for human detection and the NMPC-based controller allows for reactive tracking of a human or object of interest, even in complete darkness.
翻译:在本文中,我们提出了一种新颖的激光雷达和事件相机融合模式,用于在各种恶劣的光照条件下(如低光、无光、高对比度区域以及光源致盲的情况下)快速准确地检测机器人所在地下(SubT)环境的对象和人物。在此提出的方法中,事件相机和激光雷达的信息被融合在一起,以定位人或感兴趣的对象在机器人的本地坐标系中的位置。本地检测然后被转换成惯性坐标系并用于为非线性模型预测控制器(NMPC)设定参考值,用于在SubT环境中对人或对象进行反应式跟踪。所提出的新型融合采用深度滤波和 K 均值聚类进行激光雷达点云的处理,而事件相机中由反射激光束引起的事件则采用频率滤波和连通聚类进行处理。事件相机和激光雷达流中的聚类中心点被匹配,以定位SubT环境中安全背心和标志上的反射标记。所提出方案的有效性在实际的SubT环境(矿山)中通过Pioneer 3AT移动机器人进行了实验验证。实验结果表明了人体检测的实时性,而基于NMPC的控制器甚至可以在完全黑暗中对所感兴趣的人或物体进行反应式跟踪。