Exploring large-scale pretrained foundation models is of significant interest in computer vision because these models can be quickly transferred to many downstream tasks. This paper presents Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM. In contrast to standard encoder-decoder transformers where all decoder layers attend to encoder outputs, CoCa omits cross-attention in the first half of decoder layers to encode unimodal text representations, and cascades the remaining decoder layers which cross-attend to the image encoder for multimodal image-text representations. We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively. By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead. CoCa is pretrained end-to-end and from scratch on both web-scale alt-text data and annotated images by treating all labels simply as text, seamlessly unifying natural language supervision for representation learning. Empirically, CoCa achieves state-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks, spanning visual recognition (ImageNet, Kinetics-400/600/700, Moments-in-Time), crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps). Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1 accuracy, 90.6% with a frozen encoder and learned classification head, and new state-of-the-art 91.0% top-1 accuracy on ImageNet with a finetuned encoder.
翻译:探索大规模预设基础模型对于计算机愿景意义重大,因为这些模型可以快速传输到许多下游任务。 本文展示了对比性代码(CoCaa),这是一个最小化设计,用于预演图像文本编码解码器基础模型,同时具有对比性损失和字幕损失,从而从CLIP和SimVLM等基因化方法等对比性方法中折射模型能力。 与标准的解码解码器-解码器变异器相比,所有解码层都关注解码器输出, CoCa省在解码器前半层省略交叉关注,用于解码码码仪文本演示,Omodal-C 将其余的解码器层叠加到图像编码编码器中,同时通过Scial-deal-demodal-deal-deal-deal-lational-lational-deal-lational-deal-loral-deal-deal-deal-deal-deal-deal-deal-deal- dal- disal- disal- dal- disal- sal- dal- dal- disal- dal- dal- disal- disal- disal- sal- sal- disal-l- sal- disal- sal- disal-l-l- sal- disal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal-d-d-d-de-d-d-d-d-dal-d-d-d-l-al-al-al-al-al-al-al-d-d-d-d-al-al-ad-ad-d-d-d-l-d-d-dal-d-d-d-d-d-l-l-l-l-l-l-l-l-l-l-l-l-l-l-l-l-l-l-lal-l) 和处理,通过通过通过通过通过通过通过通过通过通过通过通过通过通过在