As panoptic segmentation provides a prediction for every pixel in input, non-standard and unseen objects systematically lead to wrong outputs. However, in safety-critical settings, robustness against out-of-distribution samples and corner cases is crucial to avoid dangerous behaviors, such as ignoring an animal or a lost cargo on the road. Since driving datasets cannot contain enough data points to properly sample the long tail of the underlying distribution, a method must deal with unknown and unseen scenarios to be deployed safely. Previous methods targeted part of this issue, by re-identifying already seen unlabeled objects. In this work, we broaden the scope proposing holistic segmentation: a task to identify and separate unseen unknown objects into instances, without learning from unknowns, while performing panoptic segmentation of known classes. We tackle this new problem with U3HS, which first finds unknowns as highly uncertain regions, then clusters the corresponding instance-aware embeddings into individual objects. By doing so, for the first time in panoptic segmentation with unknown objects, our U3HS is not trained with unknown data, thus leaving the settings unconstrained with respect to the type of objects and allowing for a holistic scene understanding. Extensive experiments and comparisons on two public datasets, namely Cityscapes and Lost&Found as a transfer, demonstrate the effectiveness of U3HS in the challenging task of holistic segmentation, with competitive closed-set panoptic segmentation performance.
翻译:由于光学部分为输入、非标准和看不见物体中的每一种像素提供了预测,因此,由于光学截面可以系统地预测输入、非标准和看不见物体中的每一个象素,导致产出出错。然而,在安全关键环境下,对分配外的样本和角落案例的稳健性对于避免危险行为至关重要,例如忽视动物或道路上丢失的货物。由于驾驶数据集不能包含足够的数据点,从而无法对底部分布的长尾进行适当的取样,因此,必须安全地部署一种方法。以前的方法是针对这一问题的一部分,通过重新确定已经见过的未贴标签对象来针对。在这项工作中,我们扩大了提出整体分割的范围:在不向未知者学习的情况下,查明和分离未知的未知对象,并将未知的未知对象分离成实例。我们用U3HS处理这个新问题,它首先发现未知的高度不稳定的区域,然后将相应的事件认知嵌入单个物体。通过重新定位,我们U3HS系统没有经过未知的数据培训,因此在不熟悉的全局性数据方面,因此使环境不难于对公众对象的对比,即深度数据类型进行对比,并允许对市内的数据进行演示。