Publications

Improving Cross-Domain Detection with Self-Supervised Learning

June 19, 2023/in Publications/by NEC Labs America

Publication Date: 6/19/2023

Event: CVPR 2023, 2nd Workshop on Learning with Limited Labelled Data for Image and Video Understanding

Reference: pp. 4745-4754, 2023

Authors: Kai Li, NEC Laboratories America, Inc., Northeastern University; Curtis Wigington, Adobe Inc.; Chris Tensmeyer, Adobe Inc.; Vlad I. Morariu, Adobe Inc.; Handong Zhao, Adobe Inc.; Varun Manjunatha, Adobe Inc.; Nikolaos Barmpalios, Adobe Inc.; Yun Fu, Northeastern University

Abstract: Cross-Domain Detection (XDD) aims to train a domain-adaptive object detector using unlabeled images from a target domain and labeled images from a source domain. Existing approaches achieve this either by aligning the feature maps or the region proposals from the two domains, or by transferring the style of source images to that of target images. In this paper, rather than proposing another method following the existing lines, we introduce a new framework complementary to existing methods. Our framework unifies some popular Self-Supervised Learning (SSL) techniques (e.g., rotation angle prediction, strong/weak data augmentation, mean teacher modeling) and adapts them to the XDD task. Our basic idea is to leverage the unsupervised nature of these SSL techniques and apply them simultaneously across domains (source and target) and models (student and teacher). These SSL techniques can thus serve as shared bridges that facilitate knowledge transfer between domains. More importantly, as these techniques are independently applied in each domain, they are complementary to existing domain alignment techniques that relies on interactions between domains (e.g., adversarial alignment). We perform extensive analyses on these SSL techniques and show that they significantly improve the performance of existing methods. In addition, we reach comparable or even better performance than the state-of-the-art methods when integrating our framework with an old well-established method.

Publication Link: https://openaccess.thecvf.com/content/CVPR2023W/L3D-IVU/html/Li_Improving_Cross-Domain_Detection_With_Self-Supervised_Learning_CVPRW_2023_paper.html

Camouflaged Object Detection with Feature Decomposition and Edge Reconstruction

June 18, 2023/in Publications/by NEC Labs America

Publication Date: 6/18/2023

Event: CVPR 2023

Reference: pp. 22046-22055, 2023

Authors: Chunming He, Tsinghua Shenzhen Graduate School; Kai Li, NEC Laboratories America, Inc.; Yachao Zhang, Tsinghua Shenzhen Graduate School; Longxiang Tang, Tsinghua Shenzhen Graduate School; Yulun Zhang, Computer Vision Lab, ETH Zurich; Zhenhua Guo, Tsinghua Shenzhen Graduate School; Xiu Li, Tsinghua Shenzhen Graduate School

Abstract: Camouflaged object detection (COD) aims to address the tough issue of identifying camouflaged objects visually blended into the surrounding backgrounds. COD is a challenging task due to the intrinsic similarity of camouflaged objects with the background, as well as their ambiguous boundaries. Existing approaches to this problem have developed various techniques to mimic the human visual system. Albeit effective in many cases, these methods still struggle when camouflaged objects are so deceptive to the vision system. In this paper, we propose the FEature Decomposition and Edge Reconstruction (FEDER) model for COD. The FEDER model addresses the intrinsic similarity of foreground and background by decomposing the features into different frequency bands using learnable wavelets. It then focuses on the most informative bands to mine subtle cues that differentiate foreground and background. To achieve this, a frequency attention module and a guidance-based feature aggregation module are developed. To combat the ambiguous boundary problem, we propose to learn an auxiliary edge reconstruction task alongside the COD task. We design an ordinary differential equation-inspired edge reconstruction module that generates exact edges. By learning the auxiliary task in conjunction with the COD task, the FEDER model can generate precise prediction maps with accurate object boundaries. Experiments show that our FEDER model significantly outperforms state-of-the-art methods with cheaper computational and memory costs.

Publication Link: https://openaccess.thecvf.com/content/CVPR2023/html/He_Camouflaged_Object_Detection_With_Feature_Decomposition_and_Edge_Reconstruction_CVPR_2023_paper.html

Exploring Compositional Visual Generation with Latent Classifier Guidance

June 18, 2023/in Publications/by NEC Labs America

Publication Date: 6/18/2023

Event: CVPR 2023 – Generative Models for Computer Vision Workshop

Reference: pp. 853-862, 2023

Authors: Changhao Shi, NEC Laboratories America, Inc., University of California, San Diego; Haomiao Ni, NEC Laboratories America, Inc., The Pennsylvania State University; Kai Li, NEC Laboratories America, Inc.; Shaobo Han, NEC Laboratories America, Inc.; Mingfu Liang, NEC Laboratories America, Inc., Northwestern University; Martin Renqiang Min, NEC Laboratories America, Inc.

Abstract: Diffusion probabilistic models have achieved enormous success in the field of image generation and manipulation. In this paper, we explore a novel paradigm of using the diffusion model and classifier guidance in the latent semantic space for compositional visual tasks. Specifically, we train latent diffusion models and auxiliary latent classifiers to facilitate non-linear navigation of latent representation generation for any pre-trained generative model with a semantic latent space. We demonstrate that such conditional generation achieved by latent classifier guidance provably maximizes a lower bound of the conditional log probability during training. To maintain the original semantics during manipulation, we introduce a new guidance term, which we show is crucial for achieving compositionality. With additional assumptions, we show that the non-linear manipulation reduces to a simple latent arithmetic approach. We show that this paradigm based on latent classifier guidance is agnostic to pre-trained generative models, and present competitive results for both image generation and sequential manipulation of real and synthetic images. Our findings suggest that latent classifier guidance is a promising approach that merits further exploration, even in the presence of other strong competing methods.

Publication Link: https://openaccess.thecvf.com/content/CVPR2023W/GCV/html/Shi_Exploring_Compositional_Visual_Generation_With_Latent_Classifier_Guidance_CVPRW_2023_paper.html

Conditional Image-to-Video Generation with Latent Flow Diffusion Models

June 18, 2023/in Publications/by NEC Labs America

Publication Date: 6/18/2023

Event: CVPR 2023

Reference: pp. 18444-18455, 2023

Authors: Haomiao Ni, NEC Laboratories America, Inc., Penn State University; Changhao Shi, NEC Laboratories America, Inc., University of California; Kai Li, NEC Laboratories America, Inc.; Sharon X. Huang, Penn State University; Martin Renqiang Min, NEC Laboratories America, Inc.

Abstract: Conditional image-to-video (cI2V) generation aims to synthesize a new plausible video starting from an image (e.g., a person’s face) and a condition (e.g., an action class label like smile). The key challenge of the cI2V task lies in the simultaneous generation of realistic spatial appearance and temporal dynamics corresponding to the given image and condition. In this paper, we propose an approach for cI2V using novel latent flow diffusion models (LFDM) that synthesize an optical flow sequence in the latent space based on the given condition to warp the given image. Compared to previous direct-synthesis-based works, our proposed LFDM can better synthesize spatial details and temporal motion by fully utilizing the spatial content of the given image and warping it in the latent space according to the generated temporally-coherent flow. The training of LFDM consists of two separate stages: (1) an unsupervised learning stage to train a latent flow auto-encoder for spatial content generation, including a flow predictor to estimate latent flow between pairs of video frames, and (2) a conditional learning stage to train a 3D-UNet-based diffusion model (DM) for temporal latent flow generation. Unlike previous DMs operating in pixel space or latent feature space that couples spatial and temporal information, the DM in our LFDM only needs to learn a low-dimensional latent flow space for motion generation, thus being more computationally efficient. We conduct comprehensive experiments on multiple datasets, where LFDM consistently outperforms prior arts. Furthermore, we show that LFDM can be easily adapted to new domains by simply finetuning the image decoder. Our code is available at https://github.com/nihaomiao/CVPR23_LFDM.

Publication Link: https://openaccess.thecvf.com/content/CVPR2023/html/Ni_Conditional_Image-to-Video_Generation_With_Latent_Flow_Diffusion_Models_CVPR_2023_paper.html

Source-Free Video Domain Adaptation with Spatial-Temporal-Historical Consistency Learning

June 18, 2023/in Publications/by NEC Labs America

Publication Date: 6/18/2023

Event: CVPR 2023

Reference: pp. 14643-14652, 2023

Authors: Kai Li, NEC Laboratories America, Inc.; Deep Patel, NEC Laboratories America, Inc.; Erik Kruus, NEC Laboratories America, Inc.; Martin Renqiang Min, NEC Laboratories America, Inc.

Abstract: Source-free domain adaptation (SFDA) is an emerging research topic that studies how to adapt a pretrained source model using unlabeled target data. It is derived from unsupervised domain adaptation but has the advantage of not requiring labeled source data to learn adaptive models. This makes it particularly useful in real-world applications where access to source data is restricted. While there has been some SFDA work for images, little attention has been paid to videos. Naively extending image-based methods to videos without considering the unique properties of videos often leads to unsatisfactory results. In this paper, we propose a simple and highly flexible method for Source-Free Video Domain Adaptation (SFVDA), which extensively exploits consistency learning for videos from spatial, temporal, and historical perspectives. Our method is based on the assumption that videos of the same action category are drawn from the same low-dimensional space, regardless of the spatio-temporal variations in the high-dimensional space that cause domain shifts. To overcome domain shifts, we simulate spatio-temporal variations by applying spatial and temporal augmentations on target videos, and encourage the model to make consistent predictions from a video and its augmented versions. Due to the simple design, our method can be applied to various SFVDA settings, and experiments show that our method achieves state-of-the-art performance for all the settings.

Publication Link: https://openaccess.thecvf.com/content/CVPR2023/html/Li_Source-Free_Video_Domain_Adaptation_With_Spatial-Temporal-Historical_Consistency_Learning_CVPR_2023_paper.html

Towards Realizing the Value of Labeled Target Samples: a Two-Stage Approach for Semi-Supervised Domain Adaptation

June 4, 2023/in Publications/by NEC Labs America

Publication Date: 6/4/2023

Event: 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023)

Reference: 99. 1-5, 2023

Authors: Mengqun Jin, Tsinghua University; Kai Li, NEC Laboratories America, Inc.; Shuyan Li, Tsinghua University; Chunming He, Tsinghua University; Xiu Li, Tsinghua University

Abstract: Semi-Supervised Domain Adaptation (SSDA) is a recently emerging research topic that extends from the widely-investigated Unsupervised Domain Adaptation (UDA) by further having a few target samples labeled, i.e., the model is trained with labeled source samples, unlabeled target samples as well as a few labeled target samples. Compared with UDA, the key to SSDA lies how to most effectively utilize the few labeled target samples. Existing SSDA approaches simply merge the few precious labeled target samples into vast labeled source samples or further align them, which dilutes the value of labeled target samples and thus still obtains a biased model. To remedy this, in this paper, we propose to decouple SSDA as an UDA problem and a semi-supervised learning problem where we first learn an UDA model using labeled source and unlabeled target samples and then adapt the learned UDA model in a semi-supervised way using labeled and unlabeled target samples. By utilizing the labeled source samples and target samples separately, the bias problem can be well mitigated. We further propose a consistency learning based mean teacher model to effectively adapt the learned UDA model using labeled and unlabeled target samples. Experiments show our approach outperforms existing methods.

Publication Link: https://ieeexplore.ieee.org/document/10094776

T-Cell Receptor Optimization with Reinforcement Learning and Mutation Polices for Precision Immunotherapy

April 16, 2023/in Publications/by NEC Labs America

Publication Date: 4/16/2023

Event: RECOMB 2023

Reference: pp. 174-191, 2023

Authors: Ziqi Chen, NEC Laboratories America, Inc.The Ohio State University; Martin Renqiang Min, NEC Laboratories America, Inc.; Hongyu Guo, National Research Council Canada; Chao Cheng, Baylor College of Medicine; Trevor Clancy, NEC Oncoimmunity As; Xia Ning, The Ohio State University

Abstract: T cells monitor the health status of cells by identifying foreign peptides displayed on their surface. T-cell receptors (TCRs), which are protein complexes found on the surface of T cells, are able to bind to these peptides. This process is known as TCR recognition and constitutes a key step for immune response. Optimizing TCR sequences for TCR recognition represents a fundamental step towards the development of personalized treatments to trigger immune responses killing cancerous or virus-infected cells. In this paper, we formulated the search for these optimized TCRs as a reinforcement learning (RL) problem and presented a framework TCRPPO with a mutation policy using proximal policy optimization. TCRPPO mutates TCRs into effective ones that can recognize given peptides. TCRPPO leverages a reward function that combines the likelihoods of mutated sequences being valid TCRs measured by a new scoring function based on deep autoencoders, with the probabilities of mutated sequences recognizing peptides from a peptide-TCR interaction predictor. We compared TCRPPO with multiple baseline methods and demonstrated that TCRPPO significantly outperforms all the baseline methods to generate positive binding and valid TCRs. These results demonstrate the potential of TCRPPO for both precision immunotherapy and peptide-recognizing TCR motif discovery.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-031-29119-7_11

Adversarial Alignment for Source Free Object Detection

February 7, 2023/in Publications/by NEC Labs America

Publication Date: 2/7/2023

Event: AAAI 2023

Reference: pp. 1-9, 2023

Authors: Qiaosong Chu, Tsinghua University; Shuyan Li, Tsinghua University; Guangyi Chen, MBZUAI; Kai Li, NEC Laboratories America, Inc.; Xiu Li, Tsinghua University

Abstract: Source-free object detection (SFOD) aims to transfer a detector pre-trained on a label-rich source domain to an unlabeled target domain without seeing source data. While most existing SFOD methods generate pseudo labels via a source-pretrained model to guide training, these pseudo labels usually contain high noises due to heavy domain discrepancy. In order to obtain better pseudo supervisions, we divide the target domain into source-similar and source-dissimilar parts and align them in the feature space by adversarial learning. Specifically, we design a detection variance-based criterion to divide the target domain. This criterion is motivated by a finding that larger detection variances denote higher recall and larger similarity to the source domain. Then we incorporate an adversarial module into a mean teacher framework to drive the feature spaces of these two subsets indistinguishable. Extensive experiments on multiple cross-domain object detection datasets demonstrate that our proposed method consistently outperforms the compared SFOD methods. Our implementation is available at https://github.com/ChuQiaosong

Publication Link: https://ojs.aaai.org/index.php/AAAI/article/view/25119

Binding Peptide Generation for MHC Class I Proteins with Deep Reinforcement Learning

January 23, 2023/in Publications/by NEC Labs America

Publication Date: 1/23/2023

Event: Bioinformatics

Reference: 39(2):1-8, 2023

Authors: Ziqi Chen, NEC Laboratories America, Inc., The Ohio State University; Baoyi Zhang, Rice University; Hongyu Guo, National Research Council Canada; Prashant Emani, Yale University; Trevor Clancy, NEC OncoImmunity AS; Chongming Jiang, Baylor College of Medicine; Mark Gerstein, Yale University; Xia Ning, The Ohio State University; Chao Cheng, Baylor College of Medicine; Martin Renqiang Min, NEC Laboratories America, Inc.

Abstract: Motivation: MHC Class I protein plays an important role in immunotherapy by presenting immunogenic peptides to anti-tumor immune cells. The repertoires of peptides for various MHC Class I proteins are distinct, which can be reflected by their diverse binding motifs. To characterize binding motifs for MHC Class I proteins, in vitro experiments have been conducted to screen peptides with high binding affinities to hundreds of given MHC Class I proteins. However, considering tens of thousands of known MHC Class I proteins, conducting in vitro experiments for extensive MHC proteins is infeasible, and thus a more efficient and scalable way to characterize binding motifs is needed.Results: We presented a de novo generation framework, coined PepPPO, to characterize binding motif for any given MHC Class I proteins via generating repertoires of peptides presented by them. PepPPO leverages a reinforcement learning agent with a mutation policy to mutate random input peptides into positive presented ones. Using PepPPO, we characterized binding motifs for around 10 000 known human MHC Class I proteins with and without experimental for the rapid screening of neoantigens at a much lower time cost than previous deep-learning methods.

Publication Link: https://academic.oup.com/bioinformatics/article/39/2/btad055/7000336

Real-time ConcealedWeapon Detection on 3D Radar Images forWalk-through Screening System

January 3, 2023/in Publications/by NEC Labs America

Publication Date: 1/3/2023

Event: WACV 2023

Reference: pp. 673-681, 2023

Authors: Nagma S. Khan, NEC Corporation; Kazumine Ogura, NEC Corporation; Eric Cosatto, NEC Laboratories America, Inc.; Masayuki Ariyoshi, NEC Corporation

Abstract: This paper presents a framework for real-time concealed weapon detection (CWD) on 3D radar images for walk-through screening systems. The walk-through screening system aims to ensure security in crowded areas by performing CWD on walking persons, hence it requires an accurate and real-time detection approach. To ensure accuracy, a weapon needs to be detected irrespective of its 3D orientation, thus we use the 3D radar images as detection input. For achieving real-time, we reformulate classic U-Net based segmentation networks to perform 3D detection tasks. Our 3D segmentation network predicts peak-shaped probability map, instead of voxel-wise masks, to enable position inference by elementary peak detection operation on the predicted map. In the peak-shaped probability map, the peak marks the weapon’s position. So, weapon detection task translates to peak detection on the probability map. A Gaussian function is used to model weapons in the probability map. We experimentally validate our approach on realistic 3D radar images obtained from a walk-through weapon screening system prototype. Extensive ablation studies verify the effectiveness of our proposed approach over existing conventional approaches. The experimental results demonstrate that our proposed approach can perform accurate and real-time CWD, thus making it suitable for practical applications of walk-through screening.

Publication Link: https://ieeexplore.ieee.org/document/10030388

Machine Learning | Publications

Improving Cross-Domain Detection with Self-Supervised Learning

Camouflaged Object Detection with Feature Decomposition and Edge Reconstruction

Exploring Compositional Visual Generation with Latent Classifier Guidance

Conditional Image-to-Video Generation with Latent Flow Diffusion Models

Source-Free Video Domain Adaptation with Spatial-Temporal-Historical Consistency Learning

Towards Realizing the Value of Labeled Target Samples: a Two-Stage Approach for Semi-Supervised Domain Adaptation

T-Cell Receptor Optimization with Reinforcement Learning and Mutation Polices for Precision Immunotherapy

Adversarial Alignment for Source Free Object Detection

Binding Peptide Generation for MHC Class I Proteins with Deep Reinforcement Learning

Real-time ConcealedWeapon Detection on 3D Radar Images forWalk-through Screening System

Contact Us

About Us

Our Pages

Read Our Blog Posts