Accurate brain tumor segmentation from Magnetic Resonance Imaging (MRI) is desirable to joint learning of multimodal images. However, in clinical practice, it is not always possible to acquire a complete set of MRIs, and the problem of missing modalities causes severe performance degradation in existing multimodal segmentation methods. In this work, we present the first attempt to exploit the Transformer for multimodal brain tumor segmentation that is robust to any combinatorial subset of available modalities. Concretely, we propose a novel multimodal Medical Transformer (mmFormer) for incomplete multimodal learning with three main components: the hybrid modality-specific encoders that bridge a convolutional encoder and an intra-modal Transformer for both local and global context modeling within each modality; an inter-modal Transformer to build and align the long-range correlations across modalities for modality-invariant features with global semantics corresponding to tumor region; a decoder that performs a progressive up-sampling and fusion with the modality-invariant features to generate robust segmentation. Besides, auxiliary regularizers are introduced in both encoder and decoder to further enhance the model's robustness to incomplete modalities. We conduct extensive experiments on the public BraTS $2018$ dataset for brain tumor segmentation. The results demonstrate that the proposed mmFormer outperforms the state-of-the-art methods for incomplete multimodal brain tumor segmentation on almost all subsets of incomplete modalities, especially by an average 19.07% improvement of Dice on tumor segmentation with only one available modality. The code is available at https://github.com/YaoZhang93/mmFormer.
翻译:从磁共振成像(MRI)中精密的脑肿瘤分解法(MRI)对于联合学习多式图像是可取的。然而,在临床实践中,并非总能获得一整套全套的MRIs,而缺少的模式问题导致现有多式分解方法的性能严重退化。在这项工作中,我们首次尝试利用变异器来进行多种脑肿瘤分解,这种变异功能与任何现有模式组合组合中的任何分解都非常强大。具体地说,我们建议采用新的不完全的多式联运医疗变异器(mm Former),用于不完全的多式联运学习,其中有三个主要组成部分:混合式特定模式的编码器,连接了同级编码的编码器和一个内部变异器,用于在每种模式中同时进行本地和全球背景建模的模型和内部变异变异器;建立和调整模式之间的长期关联,与任何组合组合组合组合组合的组合体;一个分解器,仅对不完全的公式/变异变特性进行增缩。此外,在混合的混合模式中引入了辅助的调整器,在组合模式中特别的内,特别是用于对等离子的变式的内分解,在内部分解的货币分解的内分解器和内部分解过程的极器,以显示的极器,以显示的分解法,以显示的分解,以显示的分解为不全式的方式,以推进式的方式,以推进式的方式,以推进。