Dense prediction tasks are a fundamental class of problems in computer vision. As supervised methods suffer from high pixel-wise labeling cost, a few-shot learning solution that can learn any dense task from a few labeled images is desired. Yet, current few-shot learning methods target a restricted set of tasks such as semantic segmentation, presumably due to challenges in designing a general and unified model that is able to flexibly and efficiently adapt to arbitrary tasks of unseen semantics. We propose Visual Token Matching (VTM), a universal few-shot learner for arbitrary dense prediction tasks. It employs non-parametric matching on patch-level embedded tokens of images and labels that encapsulates all tasks. Also, VTM flexibly adapts to any task with a tiny amount of task-specific parameters that modulate the matching algorithm. We implement VTM as a powerful hierarchical encoder-decoder architecture involving ViT backbones where token matching is performed at multiple feature hierarchies. We experiment VTM on a challenging variant of Taskonomy dataset and observe that it robustly few-shot learns various unseen dense prediction tasks. Surprisingly, it is competitive with fully supervised baselines using only 10 labeled examples of novel tasks (0.004% of full supervision) and sometimes outperforms using 0.1% of full supervision. Codes are available at https://github.com/GitGyun/visual_token_matching.
翻译:稠密预测任务是计算机视觉中一个根本性的问题类别。由于监督方法面临高像素标记成本,因此需要一种能够从少量标记图像中学习任何稠密任务的少样本学习解决方案。然而,当前的少样本学习方法针对一组受限的任务,例如语义分割,可能是由于面临设计一个通用和统一的模型的挑战,该模型能够灵活、高效地适应任何未见过的语义任务。本文提出了Visual Token Matching (VTM),这是一种针对任意密集预测任务的通用少样本学习器。它采用图像和标签的补丁级嵌入标记的非参数匹配,该标记封装了所有任务。此外,VTM通过微调匹配算法的一小部分任务特定参数灵活地适应于任何任务。我们将VTM实现为一种强大的分层编码器-解码器架构,其中ViT骨干网络参与多层级的特征层次结构的标记匹配。我们在Taskonomy数据集的一个具有挑战性的变体上测试了VTM,并观察到它能够稳健地少样本学习各种未见过的稠密预测任务。令人惊讶的是,它仅使用10个新任务的标记样本(全监督的0.004%)即与完全监督基线相竞争,有时甚至优于使用全监督的0.1%。代码可在https://github.com/GitGyun/visual_token_matching上获得了。