Multimodal fusion and multitask learning are two vital topics in machine learning. Despite the fruitful progress, existing methods for both problems are still brittle to the same challenge -- it remains dilemmatic to integrate the common information across modalities (resp. tasks) meanwhile preserving the specific patterns of each modality (resp. task). Besides, while they are actually closely related to each other, multimodal fusion and multitask learning are rarely explored within the same methodological framework before. In this paper, we propose Channel-Exchanging-Network (CEN) which is self-adaptive, parameter-free, and more importantly, applicable for multimodal and multitask dense image prediction. At its core, CEN adaptively exchanges channels between subnetworks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. For the application of dense image prediction, the validity of CEN is tested by four different scenarios: multimodal fusion, cycle multimodal fusion, multitask learning, and multimodal multitask learning. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of CEN compared to state-of-the-art methods. Detailed ablation studies have also been carried out, which demonstrate the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN.
翻译:尽管取得了富有成果的进展,但这两个问题的现有方法仍然难以应对同样的挑战 -- -- 在保存每种模式的具体模式的同时,将各种模式的共同信息(重写任务)整合到各种模式(重写任务)中,这仍然是两难的两难之处。此外,虽然多式联运和多任务学习实际上彼此密切相关,但在之前的同一方法框架内很少探索多模式融合和多任务学习。在本文件中,我们提议采用自我适应、无参数、更重要的是适用于多式联运和多任务密集图像预测的频道-交流网络(CEN),在其核心方面,CEN在不同模式的子网络之间进行适应性交流渠道。具体地说,频道交流进程是由单个频道的重要性自行指导的,这种重要性通过培训中Batch-Noralizlation(B)的规模来衡量。对于密集图像预测的应用,CEN的有效性由四种不同的假设进行测试:多式联运、循环混合、多任务学习和多任务多任务多任务图像预测。关于SENartimical-commilational-delening of muldal exal exal exal exal extravelopations a exal exal exal exmal exmal exal restigradustrual restigradustrual restiquedudududustrual resmal ress