In recent years, we have witnessed significant performance boost in the image captioning task based on vision-language pre-training (VLP). Scale is believed to be an important factor for this advance. However, most existing work only focuses on pre-training transformers with moderate sizes (e.g., 12 or 24 layers) on roughly 4 million images. In this paper, we present LEMON, a LargE-scale iMage captiONer, and provide the first empirical study on the scaling behavior of VLP for image captioning. We use the state-of-the-art VinVL model as our reference model, which consists of an image feature extractor and a transformer model, and scale the transformer both up and down, with model sizes ranging from 13 to 675 million parameters. In terms of data, we conduct experiments with up to 200 million image-text pairs which are automatically collected from web based on the alt attribute of the image (dubbed as ALT200M). Extensive analysis helps to characterize the performance trend as the model size and the pre-training data size increase. We also compare different training recipes, especially for training on large-scale noisy data. As a result, LEMON achieves new state of the arts on several major image captioning benchmarks, including COCO Caption, nocaps, and Conceptual Captions. We also show LEMON can generate captions with long-tail visual concepts when used in a zero-shot manner.
翻译:近年来,我们看到了基于视觉语言预培训(VLP)的图像说明任务中显著的绩效提升。 规模被认为是这一进步的一个重要因素。 然而,大多数现有工作仅侧重于大约400万图像(如12或24层)的中等规模的预培训变压器(如12或24层),在大约400万个图像上进行。 在本文中,我们展示了LEMON,一个LAE尺度的iMage aptiONer, 并首次对VLP图像说明的缩放行为进行了实证研究。 我们使用VLP模型作为我们的参考模型,由图像提取器和变压器模型组成,并将变压器向上和向下推缩,模型大小为13至6.75亿个参数。 在数据方面,我们用高达2亿张的图像文本双对进行实验,根据图像的利特属性自动从网上采集这些图像(如 AlT200M) 。 广泛分析有助于将性表现趋势描述为模型大小和前培训数据规模的模型模型, 和变压模型规模模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型模型,以及变型模型模型模型模型模型,以及变模型模型模型模型模型,以及变缩模型变缩模型,以及变变模型变模型变模型变模型变模型变模型变模型变模型变模型变模型变模型变模型变模型变模型变模型变, 以及变变模型变模型变模型变模型变模型变模型变模型变模型变模型变模型变的变模型变模型变,我们模型变的变变变变变变变, 。 我们还用模型变变变变变变, 将使用模型变变变变,在高模型变模型变模型变模型变模型变模型变模型变模型变模型变模型变模型变模型变码模型变模型变模型变模型变模型变模型变模型变模型变模型变模型变