Sensor fusion has wide applications in many domains including health care and autonomous systems. While the advent of deep learning has enabled promising multi-modal fusion of high-level features and end-to-end sensor fusion solutions, existing deep learning based sensor fusion techniques including deep gating architectures are not always resilient, leading to the issue of fusion weight inconsistency. We propose deep multi-modal sensor fusion architectures with enhanced robustness particularly under the presence of sensor failures. At the core of our gating architectures are fusion weight regularization and fusion target learning operating on auxiliary unimodal sensing networks appended to the main fusion model. The proposed regularized gating architectures outperform the existing deep learning architectures with and without gating under both clean and corrupted sensory inputs resulted from sensor failures. The demonstrated improvements are particularly pronounced when one or more multiple sensory modalities are corrupted.
翻译:虽然深层次学习的到来使高层次特征和端到端传感器融合解决方案的多模式融合大有希望,但现有的深层次学习的传感器融合技术,包括深凝层结构,并不总是具有弹性,导致聚变重量不一致的问题。我们建议了深层次的多模式传感器融合结构,其强度增强,特别是在传感器失灵的情况下。我们加固结构的核心是聚变重量的正规化和在附属于主要聚变模型的辅助单式遥感网络上进行融合的目标学习。拟议的正规化结构优于现有的深层学习结构,而且没有在清洁和腐蚀的感官投入下加固。当一种或多种感官模式被腐蚀时,所显示的改进特别明显。