We introduce a new setting of Novel Class Discovery in Semantic Segmentation (NCDSS), which aims at segmenting unlabeled images containing new classes given prior knowledge from a labeled set of disjoint classes. In contrast to existing approaches that look at novel class discovery in image classification, we focus on the more challenging semantic segmentation. In NCDSS, we need to distinguish the objects and background, and to handle the existence of multiple classes within an image, which increases the difficulty in using the unlabeled data. To tackle this new setting, we leverage the labeled base data and a saliency model to coarsely cluster novel classes for model training in our basic framework. Additionally, we propose the Entropy-based Uncertainty Modeling and Self-training (EUMS) framework to overcome noisy pseudo-labels, further improving the model performance on the novel classes. Our EUMS utilizes an entropy ranking technique and a dynamic reassignment to distill clean labels, thereby making full use of the noisy data via self-supervised learning. We build the NCDSS benchmark on the PASCAL-5$^i$ dataset. Extensive experiments demonstrate the feasibility of the basic framework (achieving an average mIoU of 49.81%) and the effectiveness of EUMS framework (outperforming the basic framework by 9.28% mIoU).
翻译:在语义分解中,我们引入了一个新的小类分类发现设置(NCDSS ), 目的是将含有以前从一组标签的脱节类中获得知识的新类的未贴标签图像进行分解。 与在图像分类中研究新型类发现的现有方法相比, 我们把重点放在更具挑战性的语义分解上。 在 NCDSS 中, 我们需要区分对象和背景, 并处理图像中存在多个类的问题, 这增加了使用未贴标签数据的难度 。 为了应对这一新设置, 我们利用标签基础数据和突出的系统模型模型模型模型模型模型来粗略地组合新类。 此外, 我们提议采用基于英特罗比的不确定性建模和自我培训(EUMS ) 框架, 以克服噪音化的假标签, 进一步改进新类中的模型性能。 我们的EURMS 使用一种星级排序技术, 和动态的重新配置, 以蒸馏干净的标签, 从而通过自我超级学习来充分利用噪音数据。 我们用NCDSS 基准为基准, IP5- $ IMIS 基础数据框架, 展示欧盟标准的可行性框架 。