Operating a robot in the open world requires a high level of robustness with respect to previously unseen environments. Optimally, the robot is able to adapt by itself to new conditions without human supervision, e.g., automatically adjusting its perception system to changing lighting conditions. In this work, we address the task of continual learning for deep learning-based monocular depth estimation and panoptic segmentation in new environments in an online manner. We introduce CoDEPS to perform continual learning involving multiple real-world domains while mitigating catastrophic forgetting by leveraging experience replay. In particular, we propose a novel domain-mixing strategy to generate pseudo-labels to adapt panoptic segmentation. Furthermore, we explicitly address the limited storage capacity of robotic systems by proposing sampling strategies for constructing a fixed-size replay buffer based on rare semantic class sampling and image diversity. We perform extensive evaluations of CoDEPS on various real-world datasets demonstrating that it successfully adapts to unseen environments without sacrificing performance on previous domains while achieving state-of-the-art results. The code of our work is publicly available at http://codeps.cs.uni-freiburg.de.
翻译:在开放世界中操作机器人需要具有对先前未见环境的高度鲁棒性。理想情况下,机器人能够自适应新环境,而无需人类监督,例如自动调整其感知系统以适应不断变化的照明条件。在这项工作中,我们针对深度学习的基于单眼的深度估计和全景分割的任务进行持续学习,并采用在线方式在新环境中进行。我们引入 CoDEPS,通过利用经验回放来降低灾难性遗忘,在涉及多个真实世界域的情况下进行持续学习。特别地,我们提出了一种新颖的领域混合策略,以生成伪标签来适应全景分割。此外,我们通过提出基于稀有语义类采样和图像多样性的抽样策略,明确解决了机器人系统的存储容量有限的问题。我们在各种实际数据集上对 CoDEPS 进行了广泛评估,证明它成功地适应了未见环境,同时又不损害以前域的性能,同时实现了最先进的结果。我们的工作代码公开在 http://codeps.cs.uni-freiburg.de 上。