A surge of interest has emerged in utilizing Transformers in diverse vision tasks owing to its formidable performance. However, existing approaches primarily focus on optimizing internal model architecture designs that often entail significant trial and error with high burdens. In this work, we propose a new paradigm dubbed Decision Stream Calibration that boosts the performance of general Vision Transformers. To achieve this, we shed light on the information propagation mechanism in the learning procedure by exploring the correlation between different tokens and the relevance coefficient of multiple dimensions. Upon further analysis, it was discovered that 1) the final decision is associated with tokens of foreground targets, while token features of foreground target will be transmitted into the next layer as much as possible, and the useless token features of background area will be eliminated gradually in the forward propagation. 2) Each category is solely associated with specific sparse dimensions in the tokens. Based on the discoveries mentioned above, we designed a two-stage calibration scheme, namely ViT-Calibrator, including token propagation calibration stage and dimension propagation calibration stage. Extensive experiments on commonly used datasets show that the proposed approach can achieve promising results. The source codes are given in the supplements.
翻译:近来,由于其惊人的性能,在各种视觉任务中利用Transformer引起了极大的兴趣。 然而,现有的方法主要侧重于优化内部模型架构设计,这往往需要大量的尝试和高负担。在这项工作中,我们提出了一种名为决策流校准(Decision Stream Calibration)的新范式,以提高通用视觉Transformer的性能。为了实现这一目标,我们探讨了学习过程中的信息传递机制,研究了不同令牌之间的关联性以及多个维度的相关系数。通过进一步的分析,发现:1)最终决策与前景目标的令牌相关联,而前景目标的令牌特征将尽可能传递到下一层,而背景区域的无用令牌特征将逐渐消失。2)每个类别仅与令牌中特定的稀疏维度相关联。基于上述发现,我们设计了一个两阶段校准方案,即ViT-Calibrator,包括令牌传播校准阶段和维度传播校准阶段。在常用的数据集上进行了大量实验证明了所提出方法的效果优异。该论文附带了源代码。