Vision-language pre-training models (VLPs) have exhibited revolutionary improvements in various vision-language tasks. In VLP, some adversarial attacks fool a model into false or absurd classifications. Previous studies addressed these attacks by fine-tuning the model or changing its architecture. However, these methods risk losing the original model's performance and are difficult to apply to downstream tasks. In particular, their applicability to other tasks has not been considered. In this study, we addressed the reduction of the impact of typographic attacks on CLIP without changing the model parameters. To achieve this, we expand the idea of ``prefix learning'' and introduce our simple yet effective method: Defense-Prefix (DP), which inserts the DP token before a class name to make words ``robust'' against typographic attacks. Our method can be easily applied to downstream tasks, such as object detection, because the proposed method is independent of the model parameters. Our method significantly improves the accuracy of classification tasks for typographic attack datasets, while maintaining the zero-shot capabilities of the model. In addition, we leverage our proposed method for object detection, demonstrating its high applicability and effectiveness. The codes and datasets will be publicly available.
翻译:视觉语言预训练模型(VLP)已在各种视觉语言任务中取得了革命性的进展。在 VLP 中,一些对抗性攻击会欺骗模型进行错误或荒谬的分类。以往的研究通过微调模型或更改其架构来解决这些攻击。然而,这些方法存在失去原始模型性能和难以适用于下游任务的风险。特别地,它们对其他任务的适用性还没有进行考虑。在本研究中,我们没有改变模型参数,而是通过扩展“前缀学习”的思想来减小排版攻击对 CLIP 的影响。我们引入了一种简单而有效的方法:防御前缀 (Defense-Prefix, DP),它在类名之前插入 DP 标记,使单词“具有鲁棒性”,抵御排版攻击。我们的方法可以轻松应用于下游任务,如目标检测,因为该方法独立于模型参数。我们的方法显著提高了排版攻击数据集的分类任务的准确性,同时保持模型的零-shot能力。此外,我们利用我们提出的方法进行目标检测,证明了其高适用性和有效性。代码和数据集将公开发布。