Multimodal imaging and correlative analysis typically require image alignment. Contrastive learning can generate representations of multimodal images, reducing the challenging task of multimodal image registration to a monomodal one. Previously, additional supervision on intermediate layers in contrastive learning has improved biomedical image classification. We evaluate if a similar approach improves representations learned for registration to boost registration performance. We explore three approaches to add contrastive supervision to the latent features of the bottleneck layer in the U-Nets encoding the multimodal images and evaluate three different critic functions. Our results show that representations learned without additional supervision on latent features perform best in the downstream task of registration on two public biomedical datasets. We investigate the performance drop by exploiting recent insights in contrastive learning in classification and self-supervised learning. We visualize the spatial relations of the learned representations by means of multidimensional scaling, and show that additional supervision on the bottleneck layer can lead to partial dimensional collapse of the intermediate embedding space.
翻译:多式成像和相关分析通常需要图像校正。对比性学习可以产生多式图像的表示,将多式联运图像注册的艰巨任务减少到单一式。以前,通过对比性学习,对中间层的额外监督改善了生物医学图像分类。我们评估类似方法是否改进了注册学的表述,以提高注册绩效。我们探索三种方法,对U-Net中瓶颈层的潜在特征增加对比性监督,将多式联运图像编码起来,并评估三种不同的批评功能。我们的结果显示,在对潜在特征没有额外监督的情况下,在两个公共生物医学数据集注册的下游任务中取得了最佳效果。我们通过利用最近在分类和自我监督学习中的对比性学习的洞见,对表现进行了调查。我们通过多层面的扩展手段,将所了解的表述的空间关系进行视觉化,并表明对瓶颈层的额外监督可能导致中间嵌入空间部分的维化崩溃。</s>