Infrared and visible image fusion can compensate for the incompleteness of single-modality imaging and provide a more comprehensive scene description based on cross-modal complementarity. Most works focus on learning the overall cross-modal features by high- and low-frequency constraints at the image level alone, ignoring the fact that cross-modal instance-level features often contain more valuable information. To fill this gap, we model cross-modal instance-level features by embedding instance information into a set of Mixture-of-Experts (MoEs) for the first time, prompting image fusion networks to specifically learn instance-level information. We propose a novel framework with instance embedded Mixture-of-Experts for infrared and visible image fusion, termed MoE-Fusion, which contains an instance embedded MoE group (IE-MoE), an MoE-Decoder, two encoders, and two auxiliary detection networks. By embedding the instance-level information learned in the auxiliary network, IE-MoE achieves specialized learning of cross-modal foreground and background features. MoE-Decoder can adaptively select suitable experts for cross-modal feature decoding and obtain fusion results dynamically. Extensive experiments show that our MoE-Fusion outperforms state-of-the-art methods in preserving contrast and texture details by learning instance-level information in cross-modal images.
翻译:红外和可见的图像融合可以弥补单一模式图像的不完全性,并提供基于跨模式互补性的更全面的场景描述。多数工作侧重于仅通过图像层面的高频和低频限制学习全方位跨模式特征,忽略了跨模式实例层面的特征往往包含更有价值的信息这一事实。为了填补这一空白,我们首次将实例信息嵌入一套混合分析图像(MoEs)中,从而模拟跨模式实例层面的特征,促进图像融合网络,以具体学习实例级信息。我们提出了一个新型框架,以实例嵌入红外和可见图像融合的混合和低频度限制,称为MOE-Fusion,其中包含一个嵌入实例的MOE组(I-MOEE)、一个MOE-Decoder、两个编码级和两个辅助检测网络。通过将实例级信息嵌入辅助网络,IE-MoE实现跨模式的跨模式和背景级图像融合网络的专业化学习。MoE-Decus-delimateal Excial-restial-demodal Excial-demodial-dal-modiversal-modistryal-modual-modistrismodual ex-modual-modistrismodal mamodual-modal-modal-modal-modal-modal mamodal-modal-modal-modal-modal-modal-modal-modismodismodal mamamamadal madal madal 能够通过通过适当的动态的动态和感化方法,在动态和感性变变的动态的动态的模型中,以获得适合的动态的动态的模型的模型取得该等的动态和感变变变变变变变变变变的模型的模型,可以可可可选择,可以选择性、调。