Foundation Models (FMs) have demonstrated unprecedented capabilities including zero-shot learning, high fidelity data synthesis, and out of domain generalization. However, as we show in this paper, FMs still have poor out-of-the-box performance on expert tasks (e.g. retrieval of car manuals technical illustrations from language queries), data for which is either unseen or belonging to a long-tail part of the data distribution of the huge datasets used for FM pre-training. This underlines the necessity to explicitly evaluate and finetune FMs on such expert tasks, arguably ones that appear the most in practical real-world applications. In this paper, we propose a first of its kind FETA benchmark built around the task of teaching FMs to understand technical documentation, via learning to match their graphical illustrations to corresponding language descriptions. Our FETA benchmark focuses on text-to-image and image-to-text retrieval in public car manuals and sales catalogue brochures. FETA is equipped with a procedure for completely automatic annotation extraction (code would be released upon acceptance), allowing easy extension of FETA to more documentation types and application domains in the future. Our automatic annotation leads to an automated performance metric shown to be consistent with metrics computed on human-curated annotations (also released). We provide multiple baselines and analysis of popular FMs on FETA leading to several interesting findings that we believe would be very valuable to the FM community, paving the way towards real-world application of FMs for practical expert tasks currently 'overlooked' by standard benchmarks focusing on common objects.
翻译:基础模型(FMs)展示了前所未有的能力,包括零发学习、高忠诚度数据合成和领域性概括,然而,正如我们在本文中显示的那样,调频模型在专家任务(如从语言查询中检索汽车手册技术插图)上仍然表现不佳,其数据要么看不见,要么属于用于调频培训前的庞大数据集数据分发的长尾部分。这突出表明,有必要明确评价和微调调调频对此类专家任务(或许是现实世界实际应用中最可行的目标)进行精细分析。在本文件中,我们提出了围绕调频教学任务(例如从语言查询中检索汽车手册技术图解图解)建立的第一个FETA基准,以了解技术文件,使其与相应的语言描述相匹配。我们的调频模型基准侧重于公共汽车手册和销售目录手册中的文本到图像到文字检索的长尾部分。 FETA将配备一个完全自动的直译调调调频基准(代码一经接受后将发布)程序,便于FETA更方便地扩展FETA到更真实的文档类型和应用领域,以便更准确地了解技术文献社区,在将来以自动化的方式分析。我们自动地展示了多度基准。