Vision Transformers (ViTs) outperforms convolutional neural networks (CNNs) in several vision tasks with its global modeling capabilities. However, ViT lacks the inductive bias inherent to convolution making it require a large amount of data for training. This results in ViT not performing as well as CNNs on small datasets like medicine and science. We experimentally found that masked autoencoders (MAE) can make the transformer focus more on the image itself, thus alleviating the data-hungry issue of ViT to some extent. Yet the current MAE model is too complex resulting in over-fitting problems on small datasets. This leads to a gap between MAEs trained on small datasets and advanced CNNs models still. Therefore, we investigated how to reduce the decoder complexity in MAE and found a more suitable architectural configuration for it with small datasets. Besides, we additionally designed a location prediction task and a contrastive learning task to introduce localization and invariance characteristics for MAE. Our contrastive learning task not only enables the model to learn high-level visual information but also allows the training of MAE's class token. This is something that most MAE improvement efforts do not consider. Extensive experiments have shown that our method shows state-of-the-art performance on standard small datasets as well as medical datasets with few samples compared to the current popular masked image modeling (MIM) and vision transformers for small datasets.The code and models are available at https://github.com/Talented-Q/SDMAE.
翻译:视觉变异器(ViTs) 在某些具有全球建模能力的视觉任务中优于神经神经网络(CNNs) 。 然而, ViT 缺乏进化所需的大量培训数据。 结果是ViT和CNNs在医药和科学等小型数据集上表现不及CNN。 我们实验发现, 遮蔽的自动变异器(MAE) 能够让变异器更多关注图像本身, 从而在一定程度上减轻ViT的数据饥饿问题。 然而, 目前的MAE 模型过于复杂, 导致小型数据集的问题过于完善。 这导致在小数据集和高级CNN模型中训练的MAE 之间出现差距。 因此, 我们调查如何减少MAE 的解码复杂度, 并找到一个更合适的结构配置, 并且用小数据集。 此外, 我们还设计了一个定位预测任务和对比式的医学学习任务, 为MAE 引入了本地化和变异性特性。 我们对比式的学习任务不仅让小模型学习高层次的视觉变异性模型, 也使得MAE 模拟的实验中显示的高级数据。