Multi-domain Semantic Segmentation on Datasets with Overlapping Classes

Deep supervised models have an unprecedented capacity to apsorb large quantities of training data. Hence, training on all available datasets appears as a feasible approach towards accurate semantic segmentation models with graceful degradation in unusual scenes. Unfortunately, different datasets often use incompatible labels. For instance, the Cityscapes road class subsumes all pixels on driving surfaces, while Vistas defines separate classes for road markings, zebra crossings etc. Such inconsistencies pose a major obstacle towards successful multi-domain learning. We address this challenge by proposing a principled technique for learning with incompatible labeling policies. Different than recent related work, our technique allows seamless training on datasets with overlapping classes. Consequently, it can learn visual concepts which are not represented as a separate class in any of the individual datasets. We evaluate our method on a collection of seven semantic segmentation datasets across four different domains. The results exceed the state of the art in multi-domain semantic segmentation.