Publications

Source-Free Domain Adaptive Fundus Image Segmentation with Class-Balanced Mean Teacher

October 9, 2023/in Publications/by NEC Labs America

Publication Date: 10/9/2023

Event: MICCAI 2023

Reference: pp.. 684-694. 2023

Authors: Longxiang Tang, University of Electronic Science and Technology of China; Kai Li, NEC Laboratories America, Inc.; Chunming He, Tsinghua Shenzhen International Graduate School; Yulun Zhang, ETH Zurich; Xiu Li, Tsinghua Shenzhen International Graduate School

Abstract: This paper studies source-free domain adaptive fundus image segmentation which aims to adapt a pretrained fundus segmentation model to a target domain using unlabeled images. This is a challenging task because it is highly risky to adapt a model only using unlabeled data. Most existing methods tackle this task mainly by designing techniques to carefully generate pseudo labels from the models predictions and use the pseudo labels to train the model. While often obtaining positive adaption effects, these methods suffer from two major issues. First, they tend to be fairly unstable – incorrect pseudo labels abruptly emerged may cause a catastrophic impact on the model. Second, they fail to consider the severe class imbalance of fundus images where the foreground (e.g., cup) region is usually very small. This paper aims to address these two issues by proposing the Class-Balanced Mean Teacher (CBMT) model. CBMT addresses the unstable issue by proposing a weak-strong augmented mean teacher learning scheme where only the teacher model generates pseudo labels from weakly augmented images to train a student model that takes strongly augmented images as input. The teacher is updated as the moving average of the instantly trained student, which could be noisy. This prevents the teacher model from being abruptly impacted by incorrect pseudo-labels. For the class imbalance issue, CBMT proposes a novel loss calibration approach to highlight foreground classes according to global statistics. Experiments show that CBMT well addresses these two issues and outperforms existing methods on multiple benchmarks.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-031-43907-0_65

Few-Shot Video Classification via Representation Fusion and Promotion Learning

October 2, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/2/2023

Event: ICCV 2023

Reference: pp. 19311-19320, 2023

Authors: Haifeng Xia, NEC Laboratories America, Inc., Tulane Univeristy; Kai Li, NEC Laboratories America, Inc.; Martin Renqiang Min, NEC Laboratories America, Inc.; Zhengming Ding, Tulane Univeristy

Abstract: Recent few-shot video classification (FSVC) works achieve promising performance by capturing similarity across support and query samples with different temporal alignment strategies or learning discriminative features via Transformer block within each episode. However, they ignore two important issues: a) It is difficult to capture rich intrinsic action semantics from a limited number of support instances within each task. b) Redundant or irrelevant frames in videos easily weaken the positive influence of discriminative frames. To address these two issues, this paper proposes a novel Representation Fusion and Promotion Learning (RFPL) mechanism with two sub-modules: meta-action learning (MAL) and reinforced image representation (RIR). Concretely, during training stage, we perform online learning for seeking a task-shared meta-action bank to enrich task-specific action representation by injecting global knowledge. Besides, we exploit reinforcement learning to obtain the importance of each frame and refine the representation. This operation maximizes the contribution of discriminative frames to further capture the similarity of support and query samples from the same category. Our RFPL framework is highly flexible that it can be integrated with many existing FSVC methods. Extensive experiments show that RFPL significantly enhances the performance of existing FSVC models when integrated with them.

Publication Link: https://openaccess.thecvf.com/content/ICCV2023/papers/Xia_Few-Shot_Video_Classification_via_Representation_Fusion_and_Promotion_Learning_ICCV_2023_paper.pdf

Degradation-Resistant Unfolding Network for Heterogeneous Image Fusion

October 2, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/2/2023

Event: ICCV 2023

Reference: pp. 12611-12621, 2023

Authors: Chunming He, Tsinghua Shenzhen Graduate School; Kai Li, NEC Laboratories America, Inc.; Guoxia Xu, Xidian University; Yulun Zhang, ETH Z¨urich; Runze Hu, Tsinghua Shenzhen Graduate School; Zhenhua Guo, Tsinghua Shenzhen Graduate School; Xiu Li, Tsinghua Shenzhen Graduate School

Abstract: Heterogeneous image fusion (HIF) aims to enhance image quality by merging complementary information of images captured by different sensors. Early model-based approaches have strong interpretability while being limited by non-adaptive feature extractors with poor generalizability. Recent deep learning-based approaches benefit the powerful generalizability of convolutional neural networks but are known for being short of interpretability. In this paper, we propose a Degradation-Resistant Unfolding Network (DeRUN) for HIF which unifies the interpretability of model-based methods and the generalizability of deep learning-based solutions. Specifically, we first propose a novel HIF model for degradation resistance and derive its optimization procedures. Then, we frame the optimization unfolding process into the proposed DeRUN for end-to-end training. To ensure the robustness and efficiency of DeRUN, we incorporate a joint constraint strategy and a lightweight partial weight sharing module. To train DeRUN, we further propose a gradient direction-based entropy loss with powerful texture representation capacity. Extensive experiments show that DeRUN significantly outperforms existing methods on four HIF tasks as well as downstream applications using cheaper computational and memory costs.

Publication Link: https://openaccess.thecvf.com/content/ICCV2023/papers/He_Degradation-Resistant_Unfolding_Network_for_Heterogeneous_Image_Fusion_ICCV_2023_paper.pdf

MSI: Maximize Support-Set Information for Few-Shot Segmentation

October 2, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/2/2023

Event: ICCV 2023

Reference: pp. 19266-19276, 2023

Authors: Seonghyeon Moon, Rutgers University; Samuel S. Sohn, Rutgers University; Honglu Zhou, NEC Laboratories America, Inc.; Sejong Yoon, The College of New Jersey; Vladimir Pavlovic, Rutgers University; Muhammad Haris Khan, Mohamed Bin Zayed University of Artificial Intelligence; Mubbasir Kapadia, Rutgers University

Abstract: FSS (Few-shot segmentation) aims to segment a target class using a small number of labeled images (support set). To extract information relevant to the target class, a dominant approach in best performing FSS methods removes background features using a support mask. We observe that this feature excision through a limiting support mask introduces an information bottleneck in several challenging FSS cases, e.g., for small targets and/or inaccurate target boundaries. To this end, we present a novel method (MSI), which maximizes the support-set information by exploiting two complementary sources of features to generate super correlation maps. We validate the effectiveness of our approach by instantiating it into three recent and strong FSS methods. Experimental results on several publicly available FSS benchmarks show that our proposed method consistently improves performance by visible margins and leads to faster convergence.

Publication Link: https://openaccess.thecvf.com/content/ICCV2023/papers/Moon_MSI_Maximize_Support-Set_Information_for_Few-Shot_Segmentation_ICCV_2023_paper.pdf

Personalized Semantics Excitation for Federated Image Classification

October 2, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/2/2023

Event: ICCV 2023

Reference: pp. 19301-19310, 2023

Authors: Haifeng Xia, Tulane Univeristy; Kai Li, NEC Laboratories America, Inc.; Zhengming Ding, Tulane Univeristy

Abstract: Federated learning casts a light on the collaboration of distributed local clients with privacy protected to attain a more generic global model. However, significant distribution shift in input/label space across different clients makes it challenging to well generalize to all clients, which motivates personalized federated learning (PFL). Existing PFL methods typically customize the local model by fine-tuning with limited local supervision and the global model regularizer, which secures local specificity but risks ruining the global discriminative knowledge. In this paper, we propose a novel Personalized Semantics Excitation (PSE) mechanism to breakthrough this limitation by exciting and fusing personalized semantics from the global model during local model customization. Specifically, PSE explores channel-wise gradient differentiation across global and local models to idetify important low-level semantics mostly from convolutional layers which are embedded into the client-specific training.In addition, PSE deploys the collaboration of global and local models to enrich high-level feature representations and facilitate the robustness of client classifier through a cross-model attention module. Extensive experiments and analysis on various image classification benchmarks demonstrate the effectiveness and advantage of our method over the state-of-the-art PFL methods.

Publication Link: https://openaccess.thecvf.com/content/ICCV2023/papers/Xia_Personalized_Semantics_Excitation_for_Federated_Image_Classification_ICCV_2023_paper.pdf

Improving Cross-Domain Detection with Self-Supervised Learning

June 19, 2023/in Publications/by NEC Labs America

Publication Date: 6/19/2023

Event: CVPR 2023, 2nd Workshop on Learning with Limited Labelled Data for Image and Video Understanding

Reference: pp. 4745-4754, 2023

Authors: Kai Li, NEC Laboratories America, Inc., Northeastern University; Curtis Wigington, Adobe Inc.; Chris Tensmeyer, Adobe Inc.; Vlad I. Morariu, Adobe Inc.; Handong Zhao, Adobe Inc.; Varun Manjunatha, Adobe Inc.; Nikolaos Barmpalios, Adobe Inc.; Yun Fu, Northeastern University

Abstract: Cross-Domain Detection (XDD) aims to train a domain-adaptive object detector using unlabeled images from a target domain and labeled images from a source domain. Existing approaches achieve this either by aligning the feature maps or the region proposals from the two domains, or by transferring the style of source images to that of target images. In this paper, rather than proposing another method following the existing lines, we introduce a new framework complementary to existing methods. Our framework unifies some popular Self-Supervised Learning (SSL) techniques (e.g., rotation angle prediction, strong/weak data augmentation, mean teacher modeling) and adapts them to the XDD task. Our basic idea is to leverage the unsupervised nature of these SSL techniques and apply them simultaneously across domains (source and target) and models (student and teacher). These SSL techniques can thus serve as shared bridges that facilitate knowledge transfer between domains. More importantly, as these techniques are independently applied in each domain, they are complementary to existing domain alignment techniques that relies on interactions between domains (e.g., adversarial alignment). We perform extensive analyses on these SSL techniques and show that they significantly improve the performance of existing methods. In addition, we reach comparable or even better performance than the state-of-the-art methods when integrating our framework with an old well-established method.

Publication Link: https://openaccess.thecvf.com/content/CVPR2023W/L3D-IVU/html/Li_Improving_Cross-Domain_Detection_With_Self-Supervised_Learning_CVPRW_2023_paper.html

Camouflaged Object Detection with Feature Decomposition and Edge Reconstruction

June 18, 2023/in Publications/by NEC Labs America

Publication Date: 6/18/2023

Event: CVPR 2023

Reference: pp. 22046-22055, 2023

Authors: Chunming He, Tsinghua Shenzhen Graduate School; Kai Li, NEC Laboratories America, Inc.; Yachao Zhang, Tsinghua Shenzhen Graduate School; Longxiang Tang, Tsinghua Shenzhen Graduate School; Yulun Zhang, Computer Vision Lab, ETH Zurich; Zhenhua Guo, Tsinghua Shenzhen Graduate School; Xiu Li, Tsinghua Shenzhen Graduate School

Abstract: Camouflaged object detection (COD) aims to address the tough issue of identifying camouflaged objects visually blended into the surrounding backgrounds. COD is a challenging task due to the intrinsic similarity of camouflaged objects with the background, as well as their ambiguous boundaries. Existing approaches to this problem have developed various techniques to mimic the human visual system. Albeit effective in many cases, these methods still struggle when camouflaged objects are so deceptive to the vision system. In this paper, we propose the FEature Decomposition and Edge Reconstruction (FEDER) model for COD. The FEDER model addresses the intrinsic similarity of foreground and background by decomposing the features into different frequency bands using learnable wavelets. It then focuses on the most informative bands to mine subtle cues that differentiate foreground and background. To achieve this, a frequency attention module and a guidance-based feature aggregation module are developed. To combat the ambiguous boundary problem, we propose to learn an auxiliary edge reconstruction task alongside the COD task. We design an ordinary differential equation-inspired edge reconstruction module that generates exact edges. By learning the auxiliary task in conjunction with the COD task, the FEDER model can generate precise prediction maps with accurate object boundaries. Experiments show that our FEDER model significantly outperforms state-of-the-art methods with cheaper computational and memory costs.

Publication Link: https://openaccess.thecvf.com/content/CVPR2023/html/He_Camouflaged_Object_Detection_With_Feature_Decomposition_and_Edge_Reconstruction_CVPR_2023_paper.html

Conditional Image-to-Video Generation with Latent Flow Diffusion Models

June 18, 2023/in Publications/by NEC Labs America

Publication Date: 6/18/2023

Event: CVPR 2023

Reference: pp. 18444-18455, 2023

Authors: Haomiao Ni, NEC Laboratories America, Inc., Penn State University; Changhao Shi, NEC Laboratories America, Inc., University of California; Kai Li, NEC Laboratories America, Inc.; Sharon X. Huang, Penn State University; Martin Renqiang Min, NEC Laboratories America, Inc.

Abstract: Conditional image-to-video (cI2V) generation aims to synthesize a new plausible video starting from an image (e.g., a person’s face) and a condition (e.g., an action class label like smile). The key challenge of the cI2V task lies in the simultaneous generation of realistic spatial appearance and temporal dynamics corresponding to the given image and condition. In this paper, we propose an approach for cI2V using novel latent flow diffusion models (LFDM) that synthesize an optical flow sequence in the latent space based on the given condition to warp the given image. Compared to previous direct-synthesis-based works, our proposed LFDM can better synthesize spatial details and temporal motion by fully utilizing the spatial content of the given image and warping it in the latent space according to the generated temporally-coherent flow. The training of LFDM consists of two separate stages: (1) an unsupervised learning stage to train a latent flow auto-encoder for spatial content generation, including a flow predictor to estimate latent flow between pairs of video frames, and (2) a conditional learning stage to train a 3D-UNet-based diffusion model (DM) for temporal latent flow generation. Unlike previous DMs operating in pixel space or latent feature space that couples spatial and temporal information, the DM in our LFDM only needs to learn a low-dimensional latent flow space for motion generation, thus being more computationally efficient. We conduct comprehensive experiments on multiple datasets, where LFDM consistently outperforms prior arts. Furthermore, we show that LFDM can be easily adapted to new domains by simply finetuning the image decoder. Our code is available at https://github.com/nihaomiao/CVPR23_LFDM.

Publication Link: https://openaccess.thecvf.com/content/CVPR2023/html/Ni_Conditional_Image-to-Video_Generation_With_Latent_Flow_Diffusion_Models_CVPR_2023_paper.html

Exploring Compositional Visual Generation with Latent Classifier Guidance

June 18, 2023/in Publications/by NEC Labs America

Publication Date: 6/18/2023

Event: CVPR 2023 – Generative Models for Computer Vision Workshop

Reference: pp. 853-862, 2023

Authors: Changhao Shi, NEC Laboratories America, Inc., University of California, San Diego; Haomiao Ni, NEC Laboratories America, Inc., The Pennsylvania State University; Kai Li, NEC Laboratories America, Inc.; Shaobo Han, NEC Laboratories America, Inc.; Mingfu Liang, NEC Laboratories America, Inc., Northwestern University; Martin Renqiang Min, NEC Laboratories America, Inc.

Abstract: Diffusion probabilistic models have achieved enormous success in the field of image generation and manipulation. In this paper, we explore a novel paradigm of using the diffusion model and classifier guidance in the latent semantic space for compositional visual tasks. Specifically, we train latent diffusion models and auxiliary latent classifiers to facilitate non-linear navigation of latent representation generation for any pre-trained generative model with a semantic latent space. We demonstrate that such conditional generation achieved by latent classifier guidance provably maximizes a lower bound of the conditional log probability during training. To maintain the original semantics during manipulation, we introduce a new guidance term, which we show is crucial for achieving compositionality. With additional assumptions, we show that the non-linear manipulation reduces to a simple latent arithmetic approach. We show that this paradigm based on latent classifier guidance is agnostic to pre-trained generative models, and present competitive results for both image generation and sequential manipulation of real and synthetic images. Our findings suggest that latent classifier guidance is a promising approach that merits further exploration, even in the presence of other strong competing methods.

Publication Link: https://openaccess.thecvf.com/content/CVPR2023W/GCV/html/Shi_Exploring_Compositional_Visual_Generation_With_Latent_Classifier_Guidance_CVPRW_2023_paper.html

Source-Free Video Domain Adaptation with Spatial-Temporal-Historical Consistency Learning

June 18, 2023/in Publications/by NEC Labs America

Publication Date: 6/18/2023

Event: CVPR 2023

Reference: pp. 14643-14652, 2023

Authors: Kai Li, NEC Laboratories America, Inc.; Deep Patel, NEC Laboratories America, Inc.; Erik Kruus, NEC Laboratories America, Inc.; Martin Renqiang Min, NEC Laboratories America, Inc.

Abstract: Source-free domain adaptation (SFDA) is an emerging research topic that studies how to adapt a pretrained source model using unlabeled target data. It is derived from unsupervised domain adaptation but has the advantage of not requiring labeled source data to learn adaptive models. This makes it particularly useful in real-world applications where access to source data is restricted. While there has been some SFDA work for images, little attention has been paid to videos. Naively extending image-based methods to videos without considering the unique properties of videos often leads to unsatisfactory results. In this paper, we propose a simple and highly flexible method for Source-Free Video Domain Adaptation (SFVDA), which extensively exploits consistency learning for videos from spatial, temporal, and historical perspectives. Our method is based on the assumption that videos of the same action category are drawn from the same low-dimensional space, regardless of the spatio-temporal variations in the high-dimensional space that cause domain shifts. To overcome domain shifts, we simulate spatio-temporal variations by applying spatial and temporal augmentations on target videos, and encourage the model to make consistent predictions from a video and its augmented versions. Due to the simple design, our method can be applied to various SFVDA settings, and experiments show that our method achieves state-of-the-art performance for all the settings.

Publication Link: https://openaccess.thecvf.com/content/CVPR2023/html/Li_Source-Free_Video_Domain_Adaptation_With_Spatial-Temporal-Historical_Consistency_Learning_CVPR_2023_paper.html

Machine Learning | Publications

Source-Free Domain Adaptive Fundus Image Segmentation with Class-Balanced Mean Teacher

Few-Shot Video Classification via Representation Fusion and Promotion Learning

Degradation-Resistant Unfolding Network for Heterogeneous Image Fusion

MSI: Maximize Support-Set Information for Few-Shot Segmentation

Personalized Semantics Excitation for Federated Image Classification

Improving Cross-Domain Detection with Self-Supervised Learning

Camouflaged Object Detection with Feature Decomposition and Edge Reconstruction

Conditional Image-to-Video Generation with Latent Flow Diffusion Models

Exploring Compositional Visual Generation with Latent Classifier Guidance

Source-Free Video Domain Adaptation with Spatial-Temporal-Historical Consistency Learning

Contact Us

About Us

Our Pages

Read Our Blog Posts