Degradation-Resistant Unfolding Network for Heterogeneous Image Fusion

Publication Date: 10/2/2023

Event: ICCV 2023

Reference: pp. 12611-12621, 2023

Authors: Chunming He, Tsinghua Shenzhen Graduate School; Kai Li, NEC Laboratories America, Inc.; Guoxia Xu, Xidian University; Yulun Zhang, ETH Z¨urich; Runze Hu, Tsinghua Shenzhen Graduate School; Zhenhua Guo, Tsinghua Shenzhen Graduate School; Xiu Li, Tsinghua Shenzhen Graduate School

Abstract: Heterogeneous image fusion (HIF) aims to enhance image quality by merging complementary information of images captured by different sensors. Early model-based approaches have strong interpretability while being limited by non-adaptive feature extractors with poor generalizability. Recent deep learning-based approaches benefit the powerful generalizability of convolutional neural networks but are known for being short of interpretability. In this paper, we propose a Degradation-Resistant Unfolding Network (DeRUN) for HIF which unifies the interpretability of model-based methods and the generalizability of deep learning-based solutions. Specifically, we first propose a novel HIF model for degradation resistance and derive its optimization procedures. Then, we frame the optimization unfolding process into the proposed DeRUN for end-to-end training. To ensure the robustness and efficiency of DeRUN, we incorporate a joint constraint strategy and a lightweight partial weight sharing module. To train DeRUN, we further propose a gradient direction-based entropy loss with powerful texture representation capacity. Extensive experiments show that DeRUN significantly outperforms existing methods on four HIF tasks as well as downstream applications using cheaper computational and memory costs.

Publication Link: