Self-distillation Augmented Masked Autoencoders for Histopathological Image Classification

Self-supervised learning (SSL) has drawn increasing attention in pathological image analysis in recent years. Compared to contrastive learning which requires careful design, masked autoencoders (MAE) building SSL from a generative paradigm probably is a simpler method. In this paper, we introduce MAE and verify the effect of visible patches for pathological image classification. Based on it, a novel SD-MAE model is proposed to enable a self-distillation augmented SSL on top of the raw MAE. Besides the reconstruction loss on masked image patches, SD-MAE further imposes the self-distillation loss on visible patches. It transfers knowledge brought by the global attention of the decoder to the encoder which only uses local attention. We apply SD-MAE on two public pathological image datasets. Experiments demonstrate that SD-MAE performs highly competitive when compared with other SSL methods. Our code will be released soon.