Unsupervised sim-to-real domain adaptation (UDA) for semantic segmentation aims to improve the real-world test performance of a model trained on simulated data. It can save the cost of manually labeling data in real-world applications such as robot vision and autonomous driving. Traditional UDA often assumes that there are abundant unlabeled real-world data samples available during training for the adaptation. However, such an assumption does not always hold in practice owing to the collection difficulty and the scarcity of the data. Thus, we aim to relieve this need on a large number of real data, and explore the one-shot unsupervised sim-to-real domain adaptation (OSUDA) and generalization (OSDG) problem, where only one real-world data sample is available. To remedy the limited real data knowledge, we first construct the pseudo-target domain by stylizing the simulated data with the one-shot real data. To mitigate the sim-to-real domain gap on both the style and spatial structure level and facilitate the sim-to-real adaptation, we further propose to use class-aware cross-domain transformers with an intermediate domain randomization strategy to extract the domain-invariant knowledge, from both the simulated and pseudo-target data. We demonstrate the effectiveness of our approach for OSUDA and OSDG on different benchmarks, outperforming the state-of-the-art methods by a large margin, 10.87, 9.59, 13.05 and 15.91 mIoU on GTA, SYNTHIA$\rightarrow$Cityscapes, Foggy Cityscapes, respectively.
翻译:用于语义分解的未经监督的SIM-SIM-Real域适应(UDA)旨在改进模拟数据培训模型的真实世界测试性性能,可以节省在机器人视觉和自主驱动等现实世界应用中人工标签数据的成本。传统UDA通常假设在为适应培训期间有大量未经标记的真实世界数据样本。然而,由于收集困难和数据稀缺,这种假设并不总是在实际中有效。因此,我们旨在减轻大量真实数据的需求,并探索在模拟数据模型中未经监督的模拟-真实域适应(OSUDA)和一般化(OSDG)问题,因为在那里只有一种真实世界数据样本。为了补救有限的真实数据知识,我们首先用一发真实数据来模拟模拟数据,从而构建假目标域域域域。为了减轻在风格和空间结构层面上的实际域域间差距,并促进SIM-im-ima-ima-Reforal-Reforal-deal-deal-deal-deal-deal-deal-deal-SUE-DG-SUDA-DA-I-SUA-I-IA-IA-I-I-I-I-I-SDA-I-I-I-IAR-SDA-IL-IL-I-I-I-I-SD-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-SD-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-I-