Following the success in advancing natural language processing and understanding, transformers are expected to bring revolutionary changes to computer vision. This work provides the first and comprehensive study on the robustness of vision transformers (ViTs) against adversarial perturbations. Tested on various white-box and transfer attack settings, we find that ViTs possess better adversarial robustness when compared with convolutional neural networks (CNNs). This observation also holds for certified robustness. We summarize the following main observations contributing to the improved robustness of ViTs: 1) Features learned by ViTs contain less low-level information and are more generalizable, which contributes to superior robustness against adversarial perturbations. 2) Introducing convolutional or tokens-to-token blocks for learning low-level features in ViTs can improve classification accuracy but at the cost of adversarial robustness. 3) Increasing the proportion of transformers in the model structure (when the model consists of both transformer and CNN blocks) leads to better robustness. But for a pure transformer model, simply increasing the size or adding layers cannot guarantee a similar effect. 4) Pre-training on larger datasets does not significantly improve adversarial robustness though it is critical for training ViTs. 5) Adversarial training is also applicable to ViT for training robust models. Furthermore, feature visualization and frequency analysis are conducted for explanation. The results show that ViTs are less sensitive to high-frequency perturbations than CNNs and there is a high correlation between how well the model learns low-level features and its robustness against different frequency-based perturbations.
翻译:在推进自然语言处理和理解的成功之后,变压器预计将给计算机愿景带来革命性的变化。 这项工作首次全面研究了视力变压器(ViTs)在对抗性扰动方面的稳健性。 在各种白箱和传输攻击设置上测试了ViTs在与进化神经网络(CNNs)相比,具有更好的对抗性强度。 这一观察也具有经认证的稳健性。 我们总结了有助于增强ViTs稳健性的以下主要观察:1) ViTs所学的特征包含的低水平信息,而且更加广泛,有助于在对抗对抗对抗性扰动性扰动性扰动时,提高高调变压器(ViTs)的稳健性。 引入ViT的变压性或代谢性模块可以提高分类的准确性能,但对于对ViT的稳健性培训来说,进行更难于稳定性分析。