Accelerated multi-modal magnetic resonance (MR) imaging is a new and effective solution for fast MR imaging, providing superior performance in restoring the target modality from its undersampled counterpart with guidance from an auxiliary modality. However, existing works simply combine the auxiliary modality as prior information, lacking in-depth investigations on the potential mechanisms for fusing different modalities. Further, they usually rely on the convolutional neural networks (CNNs), which is limited by the intrinsic locality in capturing the long-distance dependency. To this end, we propose a multi-modal transformer (MTrans), which is capable of transferring multi-scale features from the target modality to the auxiliary modality, for accelerated MR imaging. To capture deep multi-modal information, our MTrans utilizes an improved multi-head attention mechanism, named cross attention module, which absorbs features from the auxiliary modality that contribute to the target modality. Our framework provides three appealing benefits: (i) Our MTrans use an improved transformers for multi-modal MR imaging, affording more global information compared with existing CNN-based methods. (ii) A new cross attention module is proposed to exploit the useful information in each modality at different scales. The small patch in the target modality aims to keep more fine details, the large patch in the auxiliary modality aims to obtain high-level context features from the larger region and supplement the target modality effectively. (iii) We evaluate MTrans with various accelerated multi-modal MR imaging tasks, e.g., MR image reconstruction and super-resolution, where MTrans outperforms state-of-the-art methods on fastMRI and real-world clinical datasets.
翻译:多式磁共振成像(MR)是快速MR成像的一种新而有效的新解决办法,通过辅助模式的辅助模式,在恢复目标模式方面提供优异性能,从标模不足的对应方恢复目标模式,通过辅助模式提供指导;然而,现有的工程只是将辅助模式作为先前的信息,缺乏对不同模式引信潜在机制的深入调查;此外,它们通常依赖超声波神经网络(CNNs),这种网络受远程依赖的内在地点的限制。为此,我们提议了多式变异器(MTrans),它能够将目标模式的多级特征从标本模式转移到辅助模式,加速MRimimimimimation。为了获取深度多式信息的深度多式信息,我们的MTransm使用改进型变异器,与现有CNNM分辨率方法相比,提供了更多的全球信息。 (二)一个新的跨式变异式变异式变异式变式变异式变异式模型,目的是将目标型的多式MMM格式有效地用于区域,在不同的比例上,用大型变现的超式超式MM模式。