Hyperspectral imaging systems that use multispectral filter arrays (MSFA) capture only one spectral component in each pixel. Hyperspectral demosaicing is used to recover the non-measured components. While deep learning methods have shown promise in this area, they still suffer from several challenges, including limited modeling of non-local dependencies, lack of consideration of the periodic MSFA pattern that could be linked to periodic artifacts, and difficulty in recovering high-frequency details. To address these challenges, this paper proposes a novel de-mosaicing framework, the MSFA-frequency-aware Transformer network (FDM-Net). FDM-Net integrates a novel MSFA-frequency-aware multi-head self-attention mechanism (MaFormer) and a filter-based Fourier zero-padding method to reconstruct high pass components with greater difficulty and low pass components with relative ease, separately. The advantage of Maformer is that it can leverage the MSFA information and non-local dependencies present in the data. Additionally, we introduce a joint spatial and frequency loss to transfer MSFA information and enhance training on frequency components that are hard to recover. Our experimental results demonstrate that FDM-Net outperforms state-of-the-art methods with 6dB PSNR, and reconstructs high-fidelity details successfully.
翻译:多光谱滤波阵列(MSFA)捕获每个像素只有一个光谱分量的高光谱成像系统。高光谱去马赛克用于恢复未测量的分量。虽然深度学习方法在这一领域已经显示出有希望的前景,但仍然面临一些挑战,包括对非局部依赖性的建模有限,缺乏对可能与周期伪影相关的周期性MSFA模式的考虑,以及难以恢复高频细节。为了解决这些挑战,本文提出了一种新的去马赛克框架,即MSFA-frequency-aware变压器网络(FDM-Net)。FDM-Net集成了一种新颖的MSFA-frequency-aware多头自我注意力机制(MaFormer)和一种基于滤波器的傅里叶零填充方法,分别用于分别重建难度较大的高通分量和相对容易的低通分量。MaFormer的优势在于它可以利用数据中存在的MSFA信息和非局部依赖性。此外,我们引入了一种联合空间和频率的损失函数,用于传递MSFA信息,并增强对难以恢复的频率分量的训练。我们的实验结果表明,FDM-Net优于现有最先进的方法,具有6dB的PSNR,并成功地重建了高保真度的细节。