Publications

Domain Generalization Guided by Gradient Signal to Noise Ratio of Parameters

October 3, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/3/2023

Event: https://arxiv.org

Reference: https://arxiv.org/abs/2310.07361

Authors: Mateusz Michalkiewicz, University of Queensland, NEC Laboratories America, Inc.; Masoud Faraki, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.; Mahsa Baktashmotlagh, University of Queensland

Abstract: Overfitting to the source domain is a common issue in gradient-based training of deep neural networks. To compensate for the over-parameterized models, numerous regularization techniques have been introduced such as those based on dropout. While these methods achieve significant improvements on classical benchmarks such as ImageNet, their performance diminishes with the introduction of domain shift in the test set i.e. when the unseen data comes from a significantly different distribution. In this paper, we move away from the classical approach of Bernoulli sampled dropout mask construction and propose to base the selection on gradient-signal-to-noise ratio (GSNR) of network’s parameters. Specifically, at each training step, parameters with high GSNR will be discarded. Furthermore, we alleviate the burden of manually searching for the optimal dropout ratio by leveraging a meta-learning approach. We evaluate our method on standard domain generalization benchmarks and achieve competitive results on classification and face anti-spoofing problems.

Publication Link: https://arxiv.org/pdf/2310.07361.pdf

Domain Generalization Guided by Gradient Signal to Noise Ratio of Parameters

October 2, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/2/2023

Event: ICCV 2023

Reference: pp. 6177-6188, 2023

Authors: Mateusz Michalkiewicz, University of Queensland, NEC Laboratories America, Inc.; Masoud Faraki, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.; Mahsa Baktashmotlagh, University of Queensland

Abstract: Overfitting to the source domain is a common issue in gradient-based training of deep neural networks. To compensate for the over-parameterized models, numerous regularization techniques have been introduced such as those based on dropout. While these methods achieve significant improvements on classical benchmarks such as ImageNet, their performance diminishes with the introduction of domain shift in the test set i.e. when the unseen data comes from a significantly different distribution. In this paper, we move away from the classical approach of Bernoulli sampled dropout mask construction and propose to base the selection on gradient-signal-to-noise ratio (GSNR) of network’s parameters. Specifically, at each training step, parameters with high GSNR will be discarded. Furthermore, we alleviate the burden of manually searching for the optimal dropout ratio by leveraging a meta-learning approach. We evaluate our method on standard domain generalization benchmarks and achieve competitive results on classification and face anti-spoofing problems.

Publication Link: https://openaccess.thecvf.com/content/ICCV2023/html/Michalkiewicz_Domain_Generalization_Guided_by_Gradient_Signal_to_Noise_Ratio_of_ICCV_2023_paper.html

Efficient Controllable Multi-Task Architectures

October 2, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/2/2023

Event: ICCV 2023

Reference: pp. 5740-5751, 2023

Authors: Abhishek Aich, University of California, Riverside, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Amit K. Roy-Chowdhury, University of California, Riverside; Manmohan Chandraker, NEC Laboratories America, Inc.; Yumin Suh, NEC Laboratories America, Inc.

Abstract: We aim to train a multi-task model such that users can adjust the desired compute budget and relative importance of task performances after deployment, without retraining. This enables optimizing performance for dynamically varying user needs, without heavy computational overhead to train and save models for various scenarios. To this end, we propose a multi-task model consisting of a shared encoder and task-specific decoders where both encoder and decoder channel widths are slimmable. Our key idea is to control the task importance by varying the capacities of task-specific decoders, while controlling the total computational cost by jointly adjusting the encoder capacity. This improves overall accuracy by allowing a stronger encoder for a given budget, increases control over computational cost, and delivers high-quality slimmed sub-architectures based on user’s constraints. Our training strategy involves a novel `Configuration-Invariant Knowledge Distillation’ loss that enforces backbone representations to be invariant under different runtime width configurations to enhance accuracy. Further, we present a simple but effective search algorithm that translates user constraints to runtime width configurations of both the shared encoder and task decoders, for sampling the sub-architectures. The key rule for the search algorithm is to provide a larger computational budget to the higher preferred task decoder, while searching a shared encoder configuration that enhances the overall MTL performance. Various experiments on three multi-task benchmarks (PASCALContext, NYUDv2, and CIFAR100-MTL) with diverse backbone architectures demonstrate the advantage of our approach. For example, our method shows a higher controllability by 33.5% in the NYUD-v2 dataset over prior methods, while incurring much less compute cost.

Publication Link: https://openaccess.thecvf.com/content/ICCV2023/html/Aich_Efficient_Controllable_Multi-Task_Architectures_ICCV_2023_paper.html

LDP-Feat: Image Features with Local Differential Privacy

October 2, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/2/2023

Event: ICCV 2023

Reference: pp. 17580-17590, 2023

Authors: Francesco Pittaluga , NEC Laboratories America, Inc.; Bingbing Zhuang, NEC Laboratories America, Inc.

Abstract: Modern computer vision services often require users to share raw feature descriptors with an untrusted server. This presents an inherent privacy risk, as raw descriptors may be used to recover the source images from which they were extracted. To address this issue, researchers recently proposed privatizing image features by embedding them within an affine subspace containing the original feature as well as adversarial feature samples. In this paper, we propose two novel inversion attacks to show that it is possible to (approximately) recover the original image features from these embeddings, allowing us to recover privacy-critical image content. In light of such successes and the lack of theoretical privacy guarantees afforded by existing visual privacy methods, we further propose the first method to privatize image features via local differential privacy, which, unlike prior approaches, provides a guaranteed bound for privacy leakage regardless of the strength of the attacks. In addition, our method yields strong performance in visual localization as a downstream task while enjoying the privacy guarantee.

Publication Link: https://openaccess.thecvf.com/content/ICCV2023/html/Pittaluga_LDP-Feat_Image_Features_with_Local_Differential_Privacy_ICCV_2023_paper.html

OmniLabel: A Challenging Benchmark for Language-Based Object Detection

October 2, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/2/2023

Event: ICCV 2023

Reference: pp. 11953-11962, 2023

Authors: Samuel Schulter, NEC Laboratories America, Inc.; Vijay Kumar B G, NEC Laboratories America, Inc.; Yumin Suh, NEC Laboratories America, Inc.; Konstantinos M. Dafnis, Rutgers University; Zhixing Zhang, Rutgers University, NEC Laboratories America, Inc.; Shiyu Zhao, Rutgers University, NEC Laboratories America, Inc.; Dimitris Metaxas, Rutgers University

Abstract: Language-based object detection is a promising direction towards building a natural interface to describe objects in images that goes far beyond plain category names. While recent methods show great progress in that direction, proper evaluation is lacking. With OmniLabel, we propose a novel task definition, dataset, and evaluation metric. The task subsumes standard and open-vocabulary detection as well as referring expressions. With more than 30K unique object descriptions on over 25K images, OmniLabel provides a challenge benchmark with diverse and complex object descriptions in a naturally open-vocabulary setting. Moreover, a key differentiation to existing benchmarks is that our object descriptions can refer to one, multiple or even no object, hence, providing negative examples in free-form text. The proposed evaluation handles the large label space and judges performance via a modified average precision metric, which we validate by evaluating strong language-based baselines. OmniLabel indeed provides a challenging test bed for future research on language-based detection.

Publication Link: https://openaccess.thecvf.com/content/ICCV2023/html/Schulter_OmniLabel_A_Challenging_Benchmark_for_Language-Based_Object_Detection_ICCV_2023_paper.html

Improving Pseudo Labels for Open-Vocabulary Object Detection

August 2, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 8/2/2023

Event: https://arxiv.org

Reference: https://arxiv.org/abs/2308.06412

Authors: Shiyu Zhao, Rutgers University, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Long Zhao, Google Research ; Zhixing Zhang, Rutgers University, NEC Laboratories America, Inc.; Vijay Kumar BG, NEC Laboratories America, Inc.; Yumin Suh, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Dimitris Metaxas, Rutgers University

Abstract: Recent studies show promising performance in open-vocabulary object detection (OVD) using pseudo labels (PLs) from pretrained vision and language models (VLMs). However, PLs generated by VLMs are extremely noisy due to the gap between the pretraining objective of VLMs and OVD, which blocks further advances on PLs. In this paper, we aim to reduce the noise in PLs and propose a method called online Self-training And a Split-and-fusion head for OVD (SAS-Det). First, the self-training finetunes VLMs to generate high quality PLs while prevents forgetting the knowledge learned in the pretraining. Second, a split-and-fusion (SAF) head is designed to remove the noise in localization of PLs, which is usually ignored in existing methods. It also fuses complementary knowledge learned from both precise ground truth and noisy pseudo labels to boost the performance. Extensive experiments demonstrate SAS-Det is both efficient and effective. Our pseudo labeling is 3 times faster than prior methods. SAS-Det outperforms prior state-of-the-art models of the same scale by a clear margin and achieves 37.4 AP50 and 27.3 APr on novel categories of the COCO and LVIS benchmarks, respectively.

Publication Link: https://arxiv.org/pdf/2308.06412.pdf

Confidence and Dispersity Speak: Characterizing Prediction Matrix for Unsupervised Accuracy Estimation

July 23, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 7/23/2023

Event: ICML 2023

Reference: Honolulu, Hawai’i

Authors: Weijian Deng, Australian National University; Yumin Suh, NEC Laboratories America, Inc.; Stephen Gould, Australian National University; Liang Zheng, Australian National University

Abstract: This work aims to assess how well a model performs under distribution shifts without using labels. While recent methods study prediction confidence, this work reports prediction dispersity is another informative cue. Confidence reflects whether the individual prediction is certain; dispersity indicates how the overall predictions are distributed across all categories. Our key insight is that a well-performing model should give predictions with high confidence and high dispersity. That is, we need to consider both properties so as to make more accurate estimates. To this end, we use the nuclear norm that has been shown to be effective in characterizing both properties. Extensive experiments validate the effectiveness of nuclear norm for various models (e.g., ViT and ConvNeXt), different datasets (e.g., ImageNet and CUB-200), and diverse types of distribution shifts (e.g., style shift and reproduction shift). We show that the nuclear norm is more accurate and robust in accuracy estimation than existing methods. Furthermore, we validate the feasibility of other measurements (e.g., mutual information maximization) for characterizing dispersity and confidence. Lastly, we investigate the limitation of the nuclear norm, study its improved variant under severe class imbalance, and discuss potential directions.

Publication Link: https://icml.cc/virtual/2023/papers.html?filter=titles&search=Confidence+and+Dispersity+Speak:+Characterizing+Prediction+Matrix+for+Unsupervised+Accuracy+Estimation

NeurOCS: Neural NOCS Supervision for Monocular 3D Object Localization

June 18, 2023/in Publications/by NEC Labs America

Publication Date: 6/18/2023

Event: CVPR 2023

Reference: pp. 21404-21414, 2023

Authors: Zhixiang Min, Stevens Institute of Technology, NEC Laboratories America, Inc.; Bingbing Zhuang, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Buyu Liu, NEC Laboratories America, Inc.; Enrique Dunn, Stevens Institute of Technology; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Monocular 3D object localization in driving scenes is a crucial task, but challenging due to its ill-posed nature. Estimating 3D coordinates for each pixel on the object surface holds great potential as it provides dense 2D-3D geometric constraints for the underlying PnP problem. However, high-quality ground truth supervision is not available in driving scenes due to sparsity and various artifacts of Lidar data, as well as the practical infeasibility of collecting per-instance CAD models. In this work, we present NeurOCS, a framework that uses instance masks and 3D boxes as input to learn 3D object shapes by means of differentiable rendering, which further serves as supervision for learning dense object coordinates. Our approach rests on insights in learning a category-level shape prior directly from real driving scenes, while properly handling single-view ambiguities. Furthermore, we study and make critical design choices to learn object coordinates more effectively from an object-centric view. Altogether, our framework leads to new state-of-the-art in monocular 3D localization that ranks 1st on the KITTI-Object benchmark among published monocular methods.

Publication Link: https://openaccess.thecvf.com/content/CVPR2023/html/Min_NeurOCS_Neural_NOCS_Supervision_for_Monocular_3D_Object_Localization_CVPR_2023_paper.html

Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!

June 18, 2023/in Publications/by NEC Labs America

Publication Date: 6/18/2023

Event: CVPR 2023

Reference: pp. 15005-15015

Authors: Zaid Khan, Northeastern University, NEC Laboratories America, Inc.; Vijay Kumar B G,, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Yun Fu, Northeastern University; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Finetuning a large vision language model (VLM) on a target dataset after large-scale pretraining is a dominant paradigm in visual question answering (VQA). Datasets for specialized tasks such as knowledge-based VQA or VQA in non natural-image domains are orders of magnitude smaller than those for general-purpose VQA. While collecting additional labels for specialized tasks or domains can be challenging, unlabeled images are often available. We introduce SelTDA (Self-Taught Data Augmentation), a strategy for finetuning large VLMs on small-scale VQA datasets. SelTDA uses the VLM and target dataset to build a teacher model that can generate question-answer pseudolabels directly conditioned on an image alone, allowing us to pseudolabel unlabeled images. SelTDA then finetunes the initial VLM on the original dataset augmented with freshly pseudolabeled images. We describe a series of experiments showing that our self-taught data augmentation increases robustness to adversarially searched questions, counterfactual examples, and rephrasings, it improves domain generalization, and results in greater retention of numerical reasoning skills. The proposed strategy requires no additional annotations or architectural modifications, and is compatible with any modern encoder-decoder multimodal transformer. Code available at https://github.com/codezakh/SelTDA

Publication Link: https://openaccess.thecvf.com/content/CVPR2023/html/Khan_Q_How_To_Specialize_Large_Vision-Language_Models_to_Data-Scarce_VQA_CVPR_2023_paper.html

Q: How to Specialize Large Vision Language Models to Data Scarce VQA Tasks? A: Self Train on Unlabeled Images! (arXiv)

June 6, 2023/in Publications/by NEC Labs America

Publication Date: 6/6/2023

Event: arXiv

Reference: https://arxiv.org/pdf/2306.03932.pdf

Authors: Zaid Khan, NEC Laboratories America, Inc., Northeastern University, Vijay Kumar B G, , NEC Laboratories America, Inc., Samuel Schulter, NEC Laboratories America, Inc., Xiang Yu, NEC Laboratories America, Inc., Manmohan Chandrakar, NEC Laboratories America, Inc., Yun Fu, Northeastern University

Abstract: Finetuning a large vision language model (VLM) on a target dataset after large scale pretraining is a dominant paradigm in visual question answering (VQA). Datasets for specialized tasks such as knowledge based VQA or VQA in non natural image domains are orders of magnitude smaller than those for general purpose VQA. While collecting additional labels for specialized tasks or domains can be challenging, unlabeled images are often available. We introduce SelTDA (Self Taught Data Augmentation), a strategy for finetuning large VLMs on small scale VQA datasets. SelTDA uses the VLM and target dataset to build a teacher model that can generate question answer pseudolabels directly conditioned on an image alone, allowing us to pseudolabel unlabeled images. SelTDA then finetunes the initial VLM on the original dataset augmented with freshly pseudolabeled images. We describe a series of experiments showing that our self taught data augmentation increases robustness to adversarially searched questions, counterfactual examples and rephrasings, improves domain generalization, and results in greater retention of numerical reasoning skills. The proposed strategy requires no additional annotations or architectural modifications, and is compatible with any modern encoder decoder multimodal transformer. Code available at https://github.com/codezakh/SelTDA.

Publication Link: https://arxiv.org/pdf/2306.03932.pdf

OmniLabel: A Challenging Benchmark for Language Based Object Detection (arXiv)

April 4, 2023/in Publications/by NEC Labs America

Publication Date: 4/4/2023

Event: arXiv

Reference: https://arxiv.org/pdf/2304.11463.pdf

Authors: Samuel Schulter, NEC Laboratories America, Inc., Vijay Kumar B G, NEC Laboratories America, Inc., Yumin Suh, NEC Laboratories America, Inc., Konstantinos M. Dafnis, Rutgers University, Zhixing Zhang, Rutgers University, NEC Laboratories America, Inc., Shiyu Zhao, Rutgers University, NEC Laboratories America, Inc., Dimitris Metaxas, Rutgers University

Abstract: Language based object detection is a promising direction towards building a natural interface to describe objects in images that goes far beyond plain category names. While recent methods show great progress in that direction, proper evaluation is lacking. With OmniLabel, we propose a novel task definition, dataset, and evaluation metric. The task subsumes standard and open vocabulary detection as well as referring expressions. With more than 28K unique object descriptions on over 25K images, OmniLabel provides a challenging benchmark with diverse and complex object descriptions in a naturally open vocabulary setting. Moreover, a key differentiation to existing benchmarks is that our object descriptions can refer to one, multiple or even no object, hence, providing negative examples in free form text. The proposed evaluation handles the large label space and judges performance via a modified average precision metric, which we validate by evaluating strong language based baselines. OmniLabel indeed provides a challenging test bed for future research on language based detection. Visit the project website at https://www.omnilabel.org

Publication Link: https://arxiv.org/pdf/2304.11463.pdf

Split to Learn: Gradient Split for Multi-Task Human Image Analysis

January 3, 2023/in Publications/by NEC Labs America

Publication Date: 1/3/2023

Event: WACV23

Reference: pp. 4340-4346, 2023

Authors: Weijian Deng, Australian National University, NEC Laboratories America, Inc.; Yumin Suh, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Masoud Faraki, NEC Laboratories America, Inc.; Liang Zheng, Australian National University; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: This paper presents an approach to train a unified deep network that simultaneously solves multiple human-related tasks. A multi-task framework is favorable for sharing information across tasks under restricted computational resources. However, tasks not only share information but may also compete for resources and conflict with each other, making the optimization of shared parameters difficult and leading to suboptimal performance. We propose a simple but effective training scheme called GradSplit that alleviates this issue by utilizing asymmetric inter-task relations. Specifically, at each convolution module, it splits features into T groups for T tasks and trains each group only using the gradient back-propagated from the task losses with which it does not have conflicts. During training, we apply GradSplit to a series of convolution modules. As a result, each module is trained to generate a set of task-specific features using the shared features from the previous module. This enables a network to use complementary information across tasks while circumventing gradient conflicts. Experimental results show that GradSplit achieves a better accuracy-efficiency trade-off than existing methods. It minimizes accuracy drop caused by task conflicts while significantly saving compute resources in terms of both FLOPs and memory at inference. We further show that GradSplit achieves higher cross-dataset accuracy compared to single-task and other multi-task networks.

Publication Link: https://ieeexplore.ieee.org/document/10030388

Exploiting Unlabeled Data with Vision and Language Models for Object Detection

October 24, 2022/in Publications/by NEC Labs America

Publication Date: 10/24/2022

Event: ECCV 2022

Reference: LNCS 13669, pp. 159–175, 2022

Authors: Shiyu Zhao, Rutgers University, NEC Laboratories America, Inc.; Zhixing Zhang, Rutgers University, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Long Zhao, Google Research; Vijay Kumar BG, NEC Laboratories America, Inc.; Anastasis Stathopoulos, Rutgers University; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Dimitris Metaxas, Rutgers University

Abstract: Building robust and generic object detection frameworks requires scaling to larger label spaces and bigger training datasets. However, it is prohibitively costly to acquire annotations for thousands of categories at a large scale. We propose a novel method that leverages the rich semantics available in recent vision and language models to localize and classify objects in unlabeled images, effectively generating pseudo labels for object detection. Starting with a generic and class-agnostic region proposal mechanism, we use vision and language models to categorize each region of an image into any object category that is required for downstream tasks. We demonstrate the value of the generated pseudo labels in two specific tasks, open-vocabulary detection, where a model needs to generalize to unseen object categories, and semi-supervised object detection, where additional unlabeled images can be used to improve the model. Our empirical evaluation shows the effectiveness of the pseudo labels in both tasks, where we outperform competitive baselines and achieve a novel state-of-the-art for open-vocabulary object detection. Our code is available at https://github.com/xiaofeng94/VL-PLM.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-031-20077-9_10

Learning Phase Mask for Privacy-Preserving Passive Depth Estimation

October 24, 2022/in Publications/by NEC Labs America

Publication Date: 10/24/2022

Event: ECCV 2022

Reference: LNCS 13667, pp. 504–521, 2022

Authors: Zaid Tasneem, Rice University, NEC Laboratories America, Inc.; Giovanni Milione, NEC Laboratories America, Inc.; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Ashok Veeraraghavan, Rice University; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Francesco Pittaluga, NEC Laboratories America, Inc.

Abstract: With over a billion sold each year, cameras are not only becoming ubiquitous, but are driving progress in a wide range of domains such as mixed reality, robotics, and more. However, severe concerns regarding the privacy implications of camera-based solutions currently limit the range of environments where cameras can be deployed. The key question we address is: Can cameras be enhanced with a scalable solution to preserve users’ privacy without degrading their machine intelligence capabilities? Our solution is a novel end-to-end adversarial learning pipeline in which a phase mask placed at the aperture plane of a camera is jointly optimized with respect to privacy and utility objectives. We conduct an extensive design space analysis to determine operating points with desirable privacy-utility tradeoffs that are also amenable to sensor fabrication and real-world constraints. We demonstrate the first working prototype that enables passive depth estimation while inhibiting face identification.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-031-20071-7_30

Learning Semantic Segmentation from Multiple Datasets with Label Shifts

October 24, 2022/in Publications/by NEC Labs America

Publication Date: 10/24/2022

Event: ECCV 2022

Reference: LNCS 13688, pp. 20–36, 2022.

Authors: Dongwan Kim, NEC Laboratories America, Inc., Seoul National University; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Yumin Suh, NEC Laboratories America, Inc.; Masoud Faraki, NEC Laboratories America, Inc.; Sparsh Garg, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Bohyung Ha, Seoul National University

Abstract: While it is desirable to train segmentation models on an aggregation of multiple datasets, a major challenge is that the label space of each dataset may be in conflict with one another. To tackle this challenge, we propose UniSeg, an effective and model-agnostic approach to automatically train segmentation models across multiple datasets with heterogeneous label spaces, without requiring any manual relabeling efforts. Specifically, we introduce two new ideas that account for conflicting and co-occurring labels to achieve better generalization performance in unseen domains. First, we identify a gradient conflict in training incurred by mismatched label spaces and propose a class-independent binary cross-entropy loss to alleviate such label conflicts. Second, we propose a loss function that considers class-relationships across datasets for a better multi-dataset training scheme. Extensive quantitative and qualitative analyses on road-scene datasets show that UniSeg improves over multi-dataset baselines, especially on unseen datasets, e.g., achieving more than 8%p gain in IoU on KITTI. Furthermore, UniSeg achieves 39.4% IoU on the WildDash2 public benchmark, making it one of the strongest submissions in the zero-shot setting. Our project page is available at https://www.nec-labs.com/~mas/UniSeg.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-031-19815-1_2

Single-Stream Multi-level Alignment for Vision-Language Pretraining

October 24, 2022/in Publications/by NEC Labs America

Publication Date: 10/24/2022

Event: ECCV 2022

Reference: LNCS 13696, pp. 735–751, 2022

Authors: Zaid Khan, Northeastern University, NEC Laboratories America, Inc.; Vijay Kumar B G, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Raymond Fu, Northeastern University

Abstract: Self-supervised vision-language pretraining from pure images and text with a contrastive loss is effective, but ignores fine-grained alignment due to a dual-stream architecture that aligns image and text representations only on a global level. Earlier, supervised, non-contrastive methods were capable of finer-grained alignment, but required dense annotations that were not scalable. We propose a single stream architecture that aligns images and language at multiple levels: global, fine-grained patch-token, and conceptual/semantic, using two novel tasks: symmetric cross-modality reconstruction (XMM) and a pseudo-labeled key word prediction (PSL). In XMM, we mask input tokens from one modality and use cross-modal information to reconstruct the masked token, thus improving fine-grained alignment between the two modalities. In PSL, we use attention to select keywords in a caption, use a momentum encoder to recommend other important keywords that are missing from the caption but represented in the image, and then train the visual encoder to predict the presence of those keywords, helping it learn semantic concepts that are essential for grounding a textual token to an image region. We demonstrate competitive performance and improved data efficiency on image-text retrieval, grounding, visual question answering/reasoning against larger models and models trained on more data. Code and models available at zaidkhan.me/SIMLA.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-031-20059-5_42

Controllable Dynamic Multi-Task Architectures

June 19, 2022/in Publications/by NEC Labs America

Publication Date: 6/19/2022

Event: CVPR’22

Reference: pp. 10945-10954, 2022

Authors: Dripta Raychaudhuri, NEC Laboratories America, Inc., University of California, Riverside; Yumin Suh, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Masoud Faraki, NEC Laboratories America, Inc.; Amit K. Roy-Chowdhury, University of California, Riverside; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Multi-task learning commonly encounters competition for resources among tasks, specifically when model capacity is limited. This challenge motivates models which allow control over the relative importance of tasks and total compute cost during inference time. In this work, we propose such a controllable multi-task network that dynamically adjusts its architecture and weights to match the desired task preference as well as the resource constraints. In contrast to the existing dynamic multi-task approaches that adjust only the weights within a fixed architecture, our approach affords the flexibility to dynamically control the total computational cost and match the user-preferred task importance better. We propose a disentangled training of two hyper networks, by exploiting task affinity and a novel branching regularized loss, to take input preferences and accordingly predict tree-structured models with adapted weights. Experiments on three multi-task benchmarks, namely PASCAL-Context, NYU-v2, and CIFAR-100, show the efficacy of our approach. Project page is available at https://www.nec-labs.com/-mas/DYMU.

Publication Link: https://ieeexplore.ieee.org/document/9878539

Learning to Learn across Diverse Data Biases in Deep Face Recognition

June 19, 2022/in Publications/by NEC Labs America

Publication Date: 6/19/2022

Event: CVPR’22

Reference: pp. 4062-4072, 2022

Authors: Chang Liu, NEC Laboratories America, Inc., Northeastern University,; Xiang Yu, NEC Laboratories America, Inc.; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Masoud Faraki, NEC Laboratories America, Inc.; Ramin Moslemi, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego; Raymond Fu, Northeastern University

Abstract: Convolutional Neural Networks have achieved remarkable success in face recognition, in part due to the abundant availability of data. However, the data used for training CNNs is often imbalanced. Prior works largely focus on the long-tailed nature of face datasets in data volume per identity or focus on single bias variation. In this paper, we show that many bias variations such as ethnicity, head pose, occlusion and blur can jointly affect the accuracy significantly. We propose a sample level weighting approach termed Multi-variation Cosine Margin (MvCoM), to simultaneously consider the multiple variation factors, which orthogonally enhances the face recognition losses to incorporate the importance of training samples. Further, we leverage a learning to learn approach, guided by a held-out meta learning set and use an additive modeling to predict the MvCoM. Extensive experiments on challenging face recognition benchmarks demonstrate the advantages of our method in jointly handling imbalances due to multiple variations.

Publication Link: https://ieeexplore.ieee.org/document/9879488

MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation

June 19, 2022/in Publications/by NEC Labs America

Publication Date: 6/19/2022

Event: CVPR’22

Reference: pp. 16928-16937, 2022

Authors: Inkyu Shin, NEC Laboratories America, Inc., KAIST; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Bingbing Zhuang, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Buyu Liu, NEC Laboratories America, Inc.; Sparsh Garg, NEC Laboratories America, Inc.; In So Kweon, KAIST; Kuk-Jin Yoon, KAIST

Abstract: Test-time adaptation approaches have recently emerged as a practical solution for handling domain shift without access to the source domain data. In this paper, we propose and explore a new multi-modal extension of test-time adaptation for 3D semantic segmentation. We find that, directly applying existing methods usually results in performance instability at test time, because multi-modal input is not considered jointly. To design a framework that can take full advantage of multi-modality, where each modality provides regularized self-supervisory signals to other modalities, we propose two complementary modules within and across the modalities. First, Intra-modal Pseudo-label Generation (Intra-PG) is introduced to obtain reliable pseudo labels within each modality by aggregating information from two models that are both pre-trained on source data but updated with target data at different paces. Second, Inter-modal Pseudo-label Refinement (Inter-PR) adaptively selects more reliable pseudo labels from different modalities based on a proposed consistency scheme. Experiments demonstrate that our regularized pseudo labels produce stable self-learning signals in numerous multi-modal test-time adaptation scenarios for 3D semantic segmentation. Visit our project website at https://www.nec-labs.com/~mas/MM-TTA

Publication Link: https://ieeexplore.ieee.org/document/9879729

On Generalizing Beyond Domains in Cross-Domain Continual Learning

June 19, 2022/in Publications/by NEC Labs America

Publication Date: 6/19/2022

Event: CVPR’22

Reference: pp. 9255-9264, 2022

Authors: Christian Simon, NEC Laboratories America, Inc., The Australian National University, ACT, Australia; Masoud Faraki, NEC Laboratories America, Inc.; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Yumin Suh, NEC Laboratories America, Inc.; Mehrtash Harandi, Monash University, VIC, Australia; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Humans have the ability to accumulate knowledge of new tasks in varying conditions, but deep neural networks of-ten suffer from catastrophic forgetting of previously learned knowledge after learning a new task. Many recent methods focus on preventing catastrophic forgetting under the assumption of train and test data following similar distributions. In this work, we consider a more realistic scenario of continual learning under domain shifts where the model must generalize its inference to an unseen domain. To this end, we encourage learning semantically meaningful features by equipping the classifier with class similarity metrics as learning parameters which are obtained through Mahalanobis similarity computations. Learning of the backbone representation along with these extra parameters is done seamlessly in an end-to-end manner. In addition, we propose an approach based on the exponential moving average of the parameters for better knowledge distillation. We demonstrate that, to a great extent, existing continual learning algorithms fail to handle the forgetting issue under multiple distributions, while our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.

Publication Link: https://ieeexplore.ieee.org/document/9879030

Weakly But Deeply Supervised Occlusion-Reasoned Parametric Road Layouts

June 19, 2022/in Publications/by NEC Labs America

Publication Date: 6/19/2022

Event: CVPR’22

Reference: pp. 16979-16988, 2022

Authors: Buyu Liu, NEC Laboratories America, Inc.; Bingbing Zhuang, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: We propose an end-to-end network that takes a single perspective RGB image of a complex road scene as input, to produce occlusion-reasoned layouts in perspective space as well as a parametric bird’s-eye-view (BEV) space. In contrast to prior works that require dense supervision such as semantic labels in perspective view, our method only requires human annotations for parametric attributes that are cheaper and less ambiguous to obtain. To solve this challenging task, our design is comprised of modules that incorporate inductive biases to learn occlusion-reasoning, geometric transformation and semantic abstraction, where each module may be supervised by appropriately transforming the parametric annotations. We demonstrate how our design choices and proposed deep supervision help achieve meaningful representations and accurate predictions. We validate our approach on two public datasets, KITTI and NuScenes, to achieve state-of-the-art results with considerably less human supervision.

Publication Link: https://ieeexplore.ieee.org/document/9879960

MM TTA: Multi Modal Test Time Adaptation for 3D Semantic Segmentation (arXiv)

April 22, 2022/in Publications/by NEC Labs America

Publication Date: 4/22/2022

Event: arXiv

Reference: https://arxiv.org/abs/2204.12667

Authors: Inkyu Shin, KAIST, NEC Laboratories America, Inc., Yi Hsuan Tsai, NEC Laboratories America, Inc., Bingbing Zhuang, NEC Laboratories America, Inc., Samuel Schulter, NEC Laboratories America, Inc., Buyu Liu, NEC Laboratories America, Inc., Sparsh Garg, NEC Laboratories America, Inc., In So Kweon, KAIST, Kuk Jin Yoon, KAIST

Abstract: Test time adaptation approaches have recently emerged as a practical solution for handling domain shift without access to the source domain data. In this paper, we propose and explore a new multi modal extension of test time adaptation for 3D semantic segmentation. We find that directly applying existing methods usually results in performance instability at test time because multi modal input is not considered jointly. To design a framework that can take full advantage of multi modality, where each modality provides regularized self supervisory signals to other modalities, we propose two complementary modules within and across the modalities. First, Intra modal Pseudolabel Generation (Intra PG) is introduced to obtain reliable pseudo labels within each modality by aggregating information from two models that are both pre trained on source data but updated with target data at different paces. Second, Inter modal Pseudo label Refinement (Inter PR) adaptively selects more reliable pseudo labels from different modalities based on a proposed consistency scheme. Experiments demonstrate that our regularized pseudo labels produce stable self learning signals in numerous multi modal test time adaptation scenarios for 3D semantic segmentation. Visit our project website at https://www.nec labs.com/mas/MM TTA

Publication Link: https://arxiv.org/pdf/2204.12667.pdf

Exploiting Unlabeled Data with Vision and Language Models for Object Detection (arXiv)

April 15, 2022/in Publications/by NEC Labs America

Publication Date: 4/15/2022

Event: arXiv

Reference: https://arxiv.org/abs/2207.08954

Authors: Shiyu Zhao, Rutgers University, NEC Laboratories America, Inc., Zhixing Zhang, Rutgers University, Samuel Schulter, NEC Laboratories America, Inc., Long Zhao, Google Research, Vijay Kumar BG, NEC Laboratories America, Inc., Anastasis Stathopoulos, Rutgers University, Manmohan Chandraker, NEC Laboratories America, Inc., Dimitris Metaxas, Rutgers University

Abstract: Building robust and generic object detection frameworks requires scaling to larger label spaces and bigger training datasets. However, it is prohibitively costly to acquire annotations for thousands of categories at a large scale. We propose a novel method that leverages the rich semantics available in recent vision and language models to localize and classify objects in unlabeled images, effectively generating pseudo labels for object detection. Starting with a generic and class agnostic region proposal mechanism, we use vision and language models to categorize each region of an image into any object category that is required for downstream tasks. We demonstrate the value of the generated pseudo labels in two specific tasks, open vocabulary detection, where a model needs to generalize to unseen object categories, and semi supervised object detection, where additional unlabeled images can be used to improve the model. Our empirical evaluation shows the effectiveness of the pseudo labels in both tasks, where we outperform competitive baselines and achieve a novel state of the art for open vocabulary object detection. Our code is available at https://github.com/xiaofeng94/VL PLM.

Publication Link: https://arxiv.org/pdf/2207.08954.pdf

Single-Stream Multi-Level Alignment for Vision-Language Pretraining (arXiv)

March 25, 2022/in Publications/by NEC Labs America

Read Single-Stream Multi-Level Alignment for Vision-Language Pretraining (arXiv). It is a self-supervised vision language pretraining from pure images and text with a contrastive loss is effective, but ignores fine grained alignment due to a dual stream architecture that aligns image and text representations only on a global level. Earlier, supervised, non contrastive methods were capable of finer grained alignment, but required dense annotations that were not scalable. We propose a single stream architecture that aligns images and language at multiple levels: global, fine grained patch token, and conceptual/semantic, using two novel tasks: symmetric cross modality reconstruction (XMM) and a pseudo labeled key word prediction (PSL). In XMM, we mask input tokens from one modality and use cross modal information to reconstruct the masked token, thus improving fine grained alignment between the two modalities. In PSL, we use attention to select keywords in a caption, use a momentum encoder to recommend other important keywords that are missing from the caption but represented in the image, and then train the visual encoder to predict the presence of those keywords, helping it learn semantic concepts that are essential for grounding a textual token to an image region. We demonstrate competitive performance and improved data efficiency on image text retrieval, grounding, visual question answering/reasoning against larger models and models trained on more data. Code and models available at this http URL.

Controllable Dynamic Multi Task Architectures (arXiv)

March 23, 2022/in Publications/by NEC Labs America

Publication Date: 3/23/2022

Event: arXiv

Reference: https://arxiv.org/abs/2203.14949

Authors: Dripta Raychaudhuri, University of California, Riverside, NEC Laboratories America, Inc., Yumin Suh, NEC Laboratories America, Inc., Samuel Schulter, NEC Laboratories America, Inc., Xiang Yu, NEC Laboratories America, Inc., Masoud Faraki, NEC Laboratories America, Inc., Amit K. Roy Chowdhury, University of California, Riverside, Manmohan Chankdraker, NEC Laboratories America, Inc., University of California, San Diego

Abstract: Multi task learning commonly encounters competition for resources among tasks, specifically when model capacity is limited. This challenge motivates models which allow control over the relative importance of tasks and total compute cost during inference time. In this work, we propose such a controllable multi task network that dynamically adjusts its architecture and weights to match the desired task preference as well as the resource constraints. In contrast to the existing dynamic multi task approaches that adjust only the weights within a fixed architecture, our approach affords the flexibility to dynamically control the total computational cost and match the user preferred task importance better. We propose a disentangled training of two hypernetworks, by exploiting task affinity and a novel branching regularized loss, to take input preferences and accordingly predict tree structured models with adapted weights. Experiments on three multi task benchmarks, namely PASCAL Context, NYU v2, and CIFAR 100, show the efficacy of our approach. Project page is available at https://www.nec labs.com/mas/DYMU.

Publication Link: https://arxiv.org/pdf/2203.14949.pdf

Learning Semantic Segmentation from Multiple Datasets with Label Shifts (arXiv)

February 4, 2022/in Publications/by NEC Labs America

Publication Date: 2/4/2022

Event: arXiv

Reference: https://arxiv.org/abs/2202.14030

Authors: Dongwan Kim, Seoul National University, NEC Laboratories America, Inc., Yi Hsuan Tsai, NEC Laboratories America, Inc., Yumin Suh, NEC Laboratories America, Inc., Masoud Faraki, NEC Laboratories America, Inc., Sparsh Garg, NEC Laboratories America, Inc., Manmohan Chandraker, NEC Laboratories America, Inc., Bohyung Ha, Seoul National University

Abstract: With increasing applications of semantic segmentation, numerous datasets have been proposed in the past few years. Yet labeling remains expensive, thus, it is desirable to jointly train models across aggregations of datasets to enhance data volume and diversity. However, label spaces differ across datasets and may even be in conflict with one another. This paper proposes UniSeg, an effective approach to automatically train models across multiple datasets with differing label spaces, without any manual relabeling efforts. Specifically, we propose two losses that account for conflicting and co occurring labels to achieve better generalization performance in unseen domains. First, a gradient conflict in training due to mismatched label spaces is identified and a class independent binary cross entropy loss is proposed to alleviate such label conflicts. Second, a loss function that considers class relationships across datasets is proposed for a better multi dataset training scheme. Extensive quantitative and qualitative analyses on road scene datasets show that UniSeg improves over multi dataset baselines, especially on unseen datasets, e.g., achieving more than 8% gain in IoU on KITTI averaged over all the settings.

Publication Link: https://arxiv.org/pdf/2202.14030.pdf

Confidence and Dispersity Speak: Characterizing Prediction Matrix for Unsupervised Accuracy Estimation

February 2, 2022/in Publications/by NEC Labs America

Publication Date: 2/2/2022

Event: arXiv

Reference: https://arxiv.org/abs/2302.01094

Authors: Weijian Deng, Australian National University, Yumin Suh, NEC Laboratories America, Inc., Stephen Gould, Australian National University, Liang Zheng, Australian National University

Abstract: This work aims to assess how well a model performs under distribution shifts without using labels. While recent methods study prediction confidence, this work reports prediction dispersity is another informative cue. Confidence reflects whether the individual prediction is certain, dispersity indicates how the overall predictions are distributed across all categories. Our key insight is that a well performing model should give predictions with high confidence and high dispersity. That is, we need to consider both properties so as to make more accurate estimates. To this end, we use the nuclear norm that has been shown to be effective in characterizing both properties. Extensive experiments validate the effectiveness of nuclear norm for various models (e.g., ViT and ConvNeXt), different datasets (e.g., ImageNet and CUB 200), and diverse types of distribution shifts (e.g., style shift and reproduction shift). We show that the nuclear norm is more accurate and robust in accuracy estimation than existing methods. Furthermore, we validate the feasibility of other measurements (e.g., mutual information maximization) for characterizing dispersity and confidence. Lastly, we investigate the limitation of the nuclear norm, study its improved variant under severe class imbalance, and discuss potential directions.

Publication Link: https://arxiv.org/pdf/2302.01094.pdf

On Generalizing Beyond Domains in Cross Domain Continual Learning (arXiv)

November 19, 2021/in Publications/by NEC Labs America

Publication Date: 11/19/2021

Event: arXiv

Reference: https://arxiv.org/abs/2203.03970

Authors: Christian Simon, NEC Laboratories America, Inc., The Australian National University, ACT, Australia, Monash University, Masoud Faraki, NEC Laboratories America, Inc., Yi Hsuan Tsai, NEC Laboratories America, Inc., Phiar Technologies, Xiang Yu, NEC Laboratories America, Inc., Samuel Schulter, NEC Laboratories America, Inc., Yumin Suh, NEC Laboratories America, Inc., Mehrtash Harandi, Monash University, VIC, Australia, Data61, Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego

Abstract: Humans have the ability to accumulate knowledge of new tasks in varying conditions, but deep neural networks often suffer from catastrophic forgetting of previously learned knowledge after learning a new task. Many recent methods focus on preventing catastrophic forgetting under the assumption of train and test data following similar distributions. In this work, we consider a more realistic scenario of continual learning under domain shifts where the model must generalize its inference to an unseen domain. To this end, we encourage learning semantically meaningful features by equipping the classifier with class similarity metrics as learning parameters which are obtained through Mahalanobis similarity computations. Learning of the backbone representation along with these extra parameters is done seamlessly in an end to end manner. In addition, we propose an approach based on the exponential moving average of the parameters for better knowledge distillation. We demonstrate that, to a great extent, existing continual learning algorithms fail to handle the forgetting issue under multiple distributions, while our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.

Publication Link: https://arxiv.org/pdf/2203.03970.pdf

Learning Cross-Modal Contrastive Features for Video Domain Adaptation

October 11, 2021/in Publications/by NEC Labs America

Publication Date: 10/11/2021

Event: ICCV 2021, Virtual

Reference: pp. 1-10, 2021

Authors: Donghyun Kim, Boston University, NEC Laboratories America, Inc.; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Bingbing Zhuang, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Stan Sclarof, Boston University; Kate Saenko, Boston University, MIT-IBM Watson AI Lab; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Learning transferable and domain adaptive feature representations from videos is important for video-relevant tasks such as action recognition. Existing video domain adaptation methods mainly rely on adversarial feature alignment, which has been derived from the RGB image space. However, video data is usually associated with multi-modal information, e.g., RGB and optical flow, and thus it remains a challenge to design a better method that considers the cross-modal inputs under the cross-domain adaptation setting. To this end, we propose a unified framework for video domain adaptation, which simultaneously regularizes cross-modal and cross-domain feature representations. Specifically, we treat each modality in a domain as a view and leverage the contrastive learning technique with properly designed sampling strategies. As a result, our objectives regularize feature spaces, which originally lack the connection across modalities or have less alignment across domains. We conduct experiments on domain adaptive action recognition benchmark datasets, i.e., UCF, HMDB, and EPIC-Kitchens, and demonstrate the effectiveness of our components against state-of-the-art algorithms.

Publication Link: https://ieeexplore.ieee.org/document/9710306

Learning Cross modal Contrastive Features for Video Domain Adaptation (arXiv)

August 23, 2021/in Publications/by NEC Labs America

Publication Date: 8/23/2021

Event: arXiv

Reference: https://arxiv.org/abs/2108.11974

Authors: Donghyun Kim, Boston University, NEC Laboratories America, Inc., Yi Hsuan Tsai, NEC Laboratories America, Inc., Bingbing Zhuang, NEC Laboratories America, Inc., Xiang Yu, NEC Laboratories America, Inc., Stan Sclarof, Boston University, Kate Saenko, Boston University, MIT IBM Watson AI Lab, Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Learning transferable and domain adaptive feature representations from videos is important for video relevant tasks such as action recognition. Existing video domain adaptation methods mainly rely on adversarial feature alignment, which has been derived from the RGB image space. However, video data is usually associated with multi modal information, e.g., RGB and optical flow, and thus it remains a challenge to design a better method that considers the cross modal inputs under the cross domain adaptation setting. To this end, we propose a unified framework for video domain adaptation, which simultaneously regularizes cross modal and cross domain feature representations. Specifically, we treat each modality in a domain as a view and leverage the contrastive learning technique with properly designed sampling strategies. As a result, our objectives regularize feature spaces, which originally lack the connection across modalities or have less alignment across domains. We conduct experiments on domain adaptive action recognition benchmark datasets, i.e., UCF, HMDB, and EPIC Kitchens, and demonstrate the effectiveness of our components against state of the art algorithms.

Publication Link: https://arxiv.org/pdf/2108.11974.pdf

Cross-Domain Similarity Learning for Face Recognition in Unseen Domains

June 19, 2021/in Publications/by NEC Labs America

Publication Date: 6/19/2021

Event: CVPR 2021, Virtual

Reference: https://openaccess.thecvf.com/content/CVPR2021/papers/Faraki_Cross-Domain_Similarity_Learning_for_Face_Recognition_in_Unseen_Domains_CVPR_2021_paper.pdf

Authors: Masoud Faraki, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Yumin Suh, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego

Abstract: Face recognition models trained under the assumption of identical training and test distributions often suffer from poor generalization when faced with unknown variations, such as a novel ethnicity or unpredictable individual make-ups during test time. In this paper, we introduce a novel cross-domain metric learning loss, which we dub Cross-Domain Triplet (CDT) loss, to improve face recognition in unseen domains. The CDT loss encourages learning semantically meaningful features by enforcing compact feature clusters of identities from one domain, where the compactness is measured by underlying similarity metrics that belong to another training domain with different statistics. Intuitively, it discriminatively correlates explicit metrics derived from one domain, with triplet samples from another domain in a unified loss function to be minimized within a network, which leads to better alignment of the training domains. The network parameters are further enforced to learn generalized features under domain shift, in a model-agnostic learning pipeline. Unlike the recent work of Meta Face Recognition [18], our method does not require careful hard-pair sample mining and filtering strategy during training. Extensive experiments on various face recognition benchmarks show the superiority of our method in handling variations, compared to baseline and the state-of-the-art methods.

Publication Link: https://ieeexplore.ieee.org/document/9578211

Divide-and-Conquer for Lane-Aware Diverse Trajectory Prediction

June 19, 2021/in Publications/by NEC Labs America

Publication Date: 6/19/2021

Event: CVPR 2021, Virtual

Reference: https://openaccess.thecvf.com/content/CVPR2021/papers/Narayanan_Divide-and-Conquer_for_Lane-Aware_Diverse_Trajectory_Prediction_CVPR_2021_paper.pdf

Authors: Sriram Narayanan, NEC Laboratories America, Inc.; Ramin Moslemi, NEC Laboratories America, Inc.; Francesco Pittaluga, NEC Laboratories America, Inc.; Buyu Liu, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego

Abstract: Trajectory prediction is a safety-critical tool for autonomous vehicles to plan and execute actions. Our work addresses two key challenges in trajectory prediction, learning multimodal outputs, and better predictions by imposing constraints using driving knowledge. Recent methods have achieved strong performances using Multi-Choice Learning objectives like winner-takes-all (WTA) or best-of-many. But the impact of those methods in learning diverse hypotheses is under-studied as such objectives highly depend on their initialization for diversity. As our first contribution, we propose a novel Divide-And-Conquer (DAC) approach that acts as a better initialization technique to WTA objective, resulting in diverse outputs without any spurious modes. Our second contribution is a novel trajectory prediction framework called ALAN that uses existing lane centerlines as anchors to provide trajectories constrained to the input lanes. Our framework provides multi-agent trajectory outputs in a forward pass by capturing interactions through hypercolumn descriptors and incorporating scene information in the form of rasterized images and per-agent lane anchors. Experiments on synthetic and real data show that the proposed DAC captures the data distribution better compare to other WTA family of objectives. Further, we show that our ALAN approach provides on par or better performance with SOTA methods evaluated on Nuscenes urban driving benchmark.

Publication Link: https://ieeexplore.ieee.org/document/9577563

Fusing the Old with the New: Learning Relative Pose with Geometry-Guided Uncertainty

June 19, 2021/in Publications/by NEC Labs America

Publication Date: 6/19/2021

Event: CVPR 2021, Virtual

Reference: https://openaccess.thecvf.com/content/CVPR2021/papers/Zhuang_Fusing_the_Old_with_the_New_Learning_Relative_Camera_Pose_CVPR_2021_paper.pdf

Authors: Bingbing Zhuang, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Learning methods for relative camera pose estimation have been developed largely in isolation from classical geometric approaches. The question of how to integrate predictions from deep neural networks (DNNs) and solutions from geometric solvers, such as the 5-point algorithm [37], has as yet remained under-explored. In this paper, we present a novel framework that involves probabilistic fusion between the two families of predictions during network training, with a view to leveraging their complementary benefits in a learnable way. The fusion is achieved by learning the DNN un- certainty under explicit guidance by the geometric uncertainty, thereby learning to take into account the geometric solution in relation to the DNN prediction. Our network features a self-attention graph neural network, which drives the learning by enforcing strong interactions between different correspondences and potentially modeling complex relationships between points. We propose motion parmeterizations suitable for learning and show that our method achieves state-of-the-art performance on the challenging DeMoN [61] and ScanNet [8] datasets. While we focus on relative pose, we envision that our pipeline is broadly applicable for fusing classical geometry and deep learning.

Publication Link: https://ieeexplore.ieee.org/document/9578101

Weakly But Deeply Supervised Occlusion Reasoned Parametric Layouts

April 14, 2021/in Publications/by NEC Labs America

Publication Date: 4/14/2021

Event: arXiv

Reference: https://arxiv.org/abs/2104.06730

Authors: Buyu Liu, NEC Laboratories America, Inc., Bingbing Zhuang, NEC Laboratories America, Inc., Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: We propose an end to end network that takes a single perspective RGB image of a complex road scene as input, to produce occlusion reasoned layouts in perspective space as well as a parametric bird’s eye view (BEV) space. In contrast to prior works that require dense supervision such as semantic labels in perspective view, our method only requires human annotations for parametric attributes that are cheaper and less ambiguous to obtain. To solve this challenging task, our design is comprised of modules that incorporate inductive biases to learn occlusion reasoning, geometric transformation and semantic abstraction, where each module may be supervised by appropriately transforming the parametric annotations. We demonstrate how our design choices and proposed deep supervision help achieve meaningful representations and accurate predictions. We validate our approach on two public datasets, KITTI and NuScenes, to achieve state of the art results with considerably less human supervision.

Publication Link: https://arxiv.org/pdf/2104.06730.pdf

Fusing the Old with the New: Learning Relative Pose with Geometry Guided Uncertainty (arXiv)

March 31, 2021/in Publications/by NEC Labs America

Publication Date: 3/31/2021

Event: arXiv

Reference: https://arxiv.org/abs/2104.08278

Authors: Bingbing Zhuang, NEC Laboratories America, Inc., Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Learning methods for relative camera pose estimation have been developed largely in isolation from classical geometric approaches. The question of how to integrate predictions from deep neural networks (DNNs) and solutions from geometric solvers, such as the 5 point algorithm, has as yet remained under explored. In this paper, we present a novel framework that involves probabilistic fusion between the two families of predictions during network training, with a view to leveraging their complementary benefits in a learnable way. The fusion is achieved by learning the DNN uncertainty under explicit guidance by the geometric uncertainty, thereby learning to take into account the geometric solution in relation to the DNN prediction. Our network features a self attention graph neural network, which drives the learning by enforcing strong interactions between different correspondences and potentially modeling complex relationships between points. We propose motion parmeterizations suitable for learning and show that our method achieves state of the art performance on the challenging DeMoN and ScanNet datasets. While we focus on relative pose, we envision that our pipeline is broadly applicable for fusing classical geometry and deep learning.

Publication Link: https://arxiv.org/pdf/2104.08278.pdf

Divide and Conquer for Lane Aware Diverse Trajectory Prediction (arXiv)

March 31, 2021/in Publications/by NEC Labs America

Publication Date: 3/31/2021

Event: arXiv

Reference: https://arxiv.org/abs/2104.08277v1

Authors: Sriram Narayanan, NEC Laboratories America, Inc., Ramin Moslemi, NEC Laboratories America, Inc., Francesco Pittaluga, NEC Laboratories America, Inc., Buyu Liu, NEC Laboratories America, Inc., Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: Trajectory prediction is a safety critical tool for autonomous vehicles to plan and execute actions. Our work addresses two key challenges in trajectory prediction, learning multimodal outputs, and better predictions by imposing constraints using driving knowledge. Recent methods have achieved strong performances using Multi Choice Learning objectives like winner takes all (WTA) or best of many. But the impact of those methods in learning diverse hypotheses is under studied as such objectives highly depend on their initialization for diversity. As our first contribution, we propose a novel Divide And Conquer (DAC) approach that acts as a better initialization technique to WTA objective, resulting in diverse outputs without any spurious modes. Our second contribution is a novel trajectory prediction framework called ALAN that uses existing lane centerlines as anchors to provide trajectories constrained to the input lanes. Our framework provides multi agent trajectory outputs in a forward pass by capturing interactions through hypercolumn descriptors and incorporating scene information in the form of rasterized images and per agent lane anchors. Experiments on synthetic and real data show that the proposed DAC captures the data distribution better compare to other WTA family of objectives. Further, we show that our ALAN approach provides on par or better performance with SOTA methods evaluated on Nuscenes urban driving benchmark.

Publication Link: https://arxiv.org/pdf/2104.08277v1.pdf

Cross Domain Similarity Learning for Face Recognition in Unseen Domains (arXiv)

March 9, 2021/in Publications/by NEC Labs America

Publication Date: 3/9/2021

Event: arXiv

Reference: https://arxiv.org/abs/2103.07503

Authors: Masoud Faraki, NEC Laboratories America, Inc., Xiang Yu, NEC Laboratories America, Inc., Yi Hsuan Tsai, NEC Laboratories America, Inc., Yumin Suh, NEC Laboratories America, Inc., Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego

Abstract: Face recognition models trained under the assumption of identical training and test distributions often suffer from poor generalization when faced with unknown variations, such as a novel ethnicity or unpredictable individual make ups during test time. In this paper, we introduce a novel cross domain metric learning loss, which we dub Cross Domain Triplet (CDT) loss, to improve face recognition in unseen domains. The CDT loss encourages learning semantically meaningful features by enforcing compact feature clusters of identities from one domain, where the compactness is measured by underlying similarity metrics that belong to another training domain with different statistics. Intuitively, it discriminatively correlates explicit metrics derived from one domain, with triplet samples from another domain in a unified loss function to be minimized within a network, which leads to better alignment of the training domains. The network parameters are further enforced to learn generalized features under domain shift, in a model agnostic learning pipeline. Unlike the recent work of Meta Face Recognition, our method does not require careful hard pair sample mining and filtering strategy during training. Extensive experiments on various face recognition benchmarks show the superiority of our method in handling variations, compared to baseline and the state of the art methods.

Publication Link: https://arxiv.org/pdf/2103.07503.pdf

Cross-Modality 3D Object Detection

January 5, 2021/in Publications/by NEC Labs America

Publication Date: 1/5/2021

Event: WACV 2021, Virtual

Reference: pp. 1-10, 2021

Authors: Ming Zhu, Shanghai Jiao Tong University; Chao Ma, Shanghai Jiao Tong University; Pan Ji, NEC Laboratories America, Inc.; Xiaokang Yang, Shanghai Jiao Tong University

Abstract: In this paper, we focus on exploring the fusion of images and point clouds for 3D object detection in view of the complementary nature of the two modalities, i.e., images possess more semantic information while point clouds specialize in distance sensing. To this end, we present a novel two-stage multi-modal fusion network for 3D object detection, taking both binocular images and raw point clouds as input. The whole architecture facilitates two-stage fusion. The first stage aims at producing 3D proposals through sparse pointwise feature fusion. Within the first stage, we further exploit a joint anchor mechanism that enables the network to utilize 2D-3D classification and regression simultaneously for better proposal generation. The second stage works on the 2D and 3D proposal regions and fuses their dense features. In addition, we propose to use pseudo LiDAR points from stereo matching as a data augmentation method to densify the LiDAR points, as we observe that objects missed by the detection network mostly have too few points especially for far-away objects. Our experiments on the KITTI dataset show that the proposed multi-stage fusion helps the network to learn better representations.

Publication Link: https://ieeexplore.ieee.org/document/9423069

Set Augmented Triplet Loss for Video Person Re-Identification

January 5, 2021/in Publications/by NEC Labs America

Publication Date: 1/5/2021

Event: WACV 2021, Virtual

Reference: pp. 1- 10, 2021

Authors: Pengfei Fang, Australian National University/DATA61-CSIRO; Pan Ji, NEC Laboratories America, Inc.; Lars Petersson, DATA61-CSIRO; Mehrtash Harandi, Monash University

Abstract: Modern video person re-identification (re-ID) machines are often trained using a metric learning approach, supervised by a triplet loss. The triplet loss used in video re-ID is usually based on so-called clip features, each aggregated from a few frame features. In this paper, we propose to model the video clip as a set and instead study the distance between sets in the corresponding triplet loss. In contrast to the distance between clip representations, the distance between clip sets considers the pair-wise similarity of each element (i.e., frame representation) between two sets. This allows the network to directly optimize the feature representation at a frame level. Apart from the commonly-used set distance metrics (e.g., ordinary distance and Hausdorff distance), we further propose a hybrid distance metric, tailored for the set-aware triplet loss. Also, we propose a hard positive set construction strategy using the learned class prototypes in a batch. Our proposed method achieves state-of-the-art results across several standard benchmarks, demonstrating the advantages of the proposed method.

Publication Link: https://ieeexplore.ieee.org/document/9423126

Channel Recurrent Attention Networks for Video Pedestrian Retrieval

November 30, 2020/in Publications/by NEC Labs America

Publication Date: 11/30/2020

Event: ACCV 2020, Kyoto, Japan

Reference: pp. 1-20, 2020

Authors: Pengfei Fang, Australian National University, Data61; Pan Ji, NEC Laboratories America, Inc.; Jieming Zhou, Australian National University, Data61; Lars Petersson, Data61; Mehrtash T. Harandi, Monash University

Abstract: Full attention, which generates an attention value per element of the input feature maps, has been successfully demonstrated to be beneficial in visual tasks. In this work, we propose a fully attentional network, termed channel recurrent attention network, for the task of video pedestrian retrieval. The main attention unit, channel recurrent attention, identifies attention maps at the frame level by jointly leveraging spatial and channel patterns via a recurrent neural network. This channel recurrent attention is designed to build a global receptive field by recurrently receiving and learning the spatial vectors. Then, a set aggregation cell is employed to generate a compact video representation. Empirical experimental results demonstrate the superior performance of the proposed deep network, outperforming current state-of-the-art results across standard video person retrieval benchmarks, and a thorough ablation study shows the effectiveness of the proposed units.

Publication Link: https://openaccess.thecvf.com/content/ACCV2020/html/Fang_Channel_Recurrent_Attention_Networks_for_Video_Pedestrian_Retrieval_ACCV_2020_paper.html

Uncertainty Aware Physically Guided Proxy Tasks for Unseen Domain Face Anti-Spoofing

November 20, 2020/in Publications/by NEC Labs America

Publication Date: 11/20/2020

Event: arXiv

Reference: https://arxiv.org/abs/2011.14054

Authors: Junru Wu, University of Texas A&M University, NEC Laboratories America, Inc., Xiang Yu, NEC Laboratories America, Inc., Buyu Liu, NEC Laboratories America, Inc., Atlas Wang, University of Texas A&M University, Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Face anti spoofing (FAS) seeks to discriminate genuine faces from fake ones arising from any type of spoofing attack. Due to the wide varieties of attacks, it is implausible to obtain training data that spans all attack types. We propose to leverage physical cues to attain better generalization on unseen domains. As a specific demonstration, we use physically guided proxy cues such as depth, reflection, and material to complement our main anti spoofing (a.k.a liveness detection) task, with the intuition that genuine faces across domains have consistent face like geometry, minimal reflection, and skin material. We introduce a novel uncertainty aware attention scheme that independently learns to weigh the relative contributions of the main and proxy tasks, preventing the over confident issue with traditional attention modules. Further, we propose attribute assisted hard negative mining to disentangle liveness irrelevant features with liveness features during learning. We evaluate extensively on public benchmarks with intra dataset and inter dataset protocols. Our method achieves the superior performance especially in unseen domain generalization for FAS.

Publication Link: https://arxiv.org/pdf/2011.14054.pdf

Voting Based Approaches For Differentially Private Federated Learning

October 6, 2020/in Publications/by NEC Labs America

Publication Date: 10/6/2020

Event: arXiv

Reference: https://arxiv.org/abs/2010.04851

Authors: Yuqing Zhu, University of California, Santa Barbara, NEC Laboratories America, Inc., Xiang Yu, NEC Laboratories America, Inc., Yi Hsuan Tsai, NEC Laboratories America, Inc., Francesco Pittaluga, NEC Laboratories America, Inc., Masoud Faraki, NEC Laboratories America, Inc., Manmohan Chandraker, NEC Laboratories America, Inc., Yu Xiang Wang, University of California, Santa Barbara

Abstract: Differentially Private Federated Learning (DPFL) is an emerging field with many applications. Gradient averaging based DPFL methods require costly communication rounds and hardly work with large capacity models, due to the explicit dimension dependence in its added noise. In this work, inspired by knowledge transfer non federated privacy learning from Papernot et al.(2017, 2018), we design two new DPFL schemes, by voting among the data labels returned from each local model, instead of averaging the gradients, which avoids the dimension dependence and significantly reduces the communication cost. Theoretically, by applying secure multi party computation, we could exponentially amplify the (data dependent) privacy guarantees when the margin of the voting scores are large. Extensive experiments show that our approaches significantly improve the privacy utility trade off over the state of the arts in DPFL.

Publication Link: https://arxiv.org/pdf/2010.04851.pdf

Adaptation Across Extreme Variations using Unlabeled Bridges

September 7, 2020/in Publications/by NEC Labs America

Publication Date: 9/7/2020

Event: BMVC’20, Manchester, UK

Reference: pp. 1-13, 2020

Authors: Shuyang Dai, Duke University, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Lawrence Carin, Duke University; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: We tackle an unsupervised domain adaptation problem for which the domain discrepancy between labeled source and unlabeled target domains is large, due to many factors of inter- and intra-domain variation. While deep domain adaptation methods have been realized by reducing the domain discrepancy, these are difficult to apply when domains are significantly different. We propose to decompose domain discrepancy into multiple but smaller, and thus easier to minimize, discrepancies by introducing unlabeled bridging domains that connect the source and target domains. We realize our proposed approach through an extension of the domain adversarial neural network with multiple discriminators, each of which accounts for reducing discrepancies between unlabeled (bridge, target) domains and a mix of all precedent domains including source. We validate the effectiveness of our method on several adaptation tasks including object recognition and semantic segmentation.

Publication Link: https://www.bmvc2020-conference.com/assets/papers/0186.pdf

Supplemental Publication Link: https://www.bmvc2020-conference.com/assets/supp/0186_supp.zip

Pseudo RGB-D for Self-Improving Monocular SLAM and Depth Prediction

August 28, 2020/in Publications/by NEC Labs America

Publication Date: 8/28/2020

Event: ECCV 2020 – The 16th European Conference on Computer Vision, Glasgow, UK

Reference: https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123560426.pdf

Authors: Lokender Tiwari, NEC Laboratories America, Inc.; Indraprastha Institute of Information Technology; Pan Ji, NEC Laboratories America, Inc.; Quoc-Huy Tran, NEC Laboratories America, Inc.; Bingbing Zhuang, NEC Laboratories America, Inc.; Saket Anand, Indraprastha Institute of Information Technology; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Classical monocular Simultaneous Localization And Mapping (SLAM) and the recently emerging convolutional neural networks (CNNs) for monocular depth prediction represent two largely disjoint approaches towards building a 3D map of the surrounding environment. In this paper, we demonstrate that the coupling of these two by leveraging the strengths of each mitigates the other’s shortcomings. Specifically, we propose a joint narrow and wide baseline based self-improving framework, where on the one hand the CNN-predicted depth is leveraged to perform $ extit(Unknown sysvar: (pseudo))$ RGB-D feature-based SLAM, leading to better accuracy and robustness than the monocular RGB SLAM baseline. On the other hand, the bundle-adjusted 3D scene structures and camera poses from the more principled geometric SLAM are injected back into the depth network through novel wide baseline losses proposed for improving the depth prediction network, which then continues to contribute towards better pose and 3D structure estimation in the next iteration. We emphasize that our framework only requires $ extit(Unknown sysvar: ( unlabeled monocular))$ videos in both training and inference stages, and yet is able to outperform state-of-the-art self-supervised $ extit(Unknown sysvar: (monocular))$ and $ extit(Unknown sysvar: (stereo))$ depth prediction networks (e.g, Monodepth2) and feature based monocular SLAM system (i.e, ORB-SLAM). Extensive experiments on KITTI and TUM RGB-D datasets verify the superiority of our self-improving geometry-CNN framework.

Publication Link: https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/1363_ECCV_2020_paper.php

Domain Adaptive Semantic Segmentation using Weak Labels

August 23, 2020/in Publications/by NEC Labs America

Publication Date: 8/23/2020

Event: ECCV 2020 – The 16th European Conference on Computer Vision, Glasgow, UK

Reference: https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123540545.pdf

Authors: Sujoy Paul, UC Riverside, NEC Laboratories America, Inc.; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Amit K. Roy-Chowdhury, UC Riverside, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: We propose a novel framework for domain adaptation in semantic segmentation with image-level weak labels in the target domain. The weak labels may be obtained based on a model prediction for unsupervised domain adaptation (UDA), or from a human oracle in a new weakly-supervised domain adaptation (WDA) paradigm for semantic segmentation. Using weak labels is both practical and useful, since (i) collecting image-level target annotations is comparably cheap in WDA and incurs no cost in UDA, and (ii) it opens the opportunity for category-wise domain alignment. Our framework uses weak labels to enable the interplay between feature alignment and pseudo-labeling, improving both in the process of domain adaptation. Specifically, we develop a weak-label classification module to enforce the network to attend to certain categories, and then use such training signals to guide the proposed category-wise alignment method. In experiments, we show considerable improvements with respect to the existing state-of-the-arts in UDA and present a new benchmark in the WDA setting.

Publication Link: https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/896_ECCV_2020_paper.php

Image Stitching and Rectification for Hand-Held Cameras

August 23, 2020/in Publications/by NEC Labs America

Publication Date: 8/23/2020

Event: ECCV 2020 – The 16th European Conference on Computer Vision, Glasgow, UK

Reference: https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123520239.pdf

Authors: Bingbing Zhuang, NEC Laboratories America, Inc.; Quoc-Huy Tran, NEC Laboratories America, Inc.

Abstract: In this paper, we derive a new differential homography that can account for the scanline-varying camera poses in Rolling Shutter (RS) cameras, and demonstrate its application to carry out RS-aware image stitching and rectification at one stroke. Despite the high complexity of RS geometry, we focus in this paper on a special yet common input — two consecutive frames from a video stream, wherein the inter-frame motion is restricted from being arbitrarily large. This allows us to adopt simpler differential motion model, leading to a straightforward and practical minimal solver. To deal with non-planar scene and camera parallax in stitching, we further propose an RS-aware spatially-varying homogarphy field in the principle of As-Projective-As-Possible (APAP). We show superior performance over state-of-the-art methods both in RS image stitching and rectification, especially for images captured by hand-held shaking cameras.

Publication Link: https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/184_ECCV_2020_paper.php

Improving Face Recognition by Clustering Unlabeled Faces in the Wild

August 23, 2020/in Publications/by NEC Labs America

Publication Date: 8/23/2020

Event: ECCV 2020 – The 16th European Conference on Computer Vision, Glasgow, UK

Reference: https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123690120.pdf

Authors: Aruni RoyChowdhury, University of Massachusetts Amherst, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Erik Learned-Miller, University of Massachusetts Amherst; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: While deep face recognition has benefited significantly from large-scale labeled data, current research is focused on leveraging unlabeled data to further boost performance, reducing the cost of human annotation. Prior work has mostly been in controlled settings, where the labeled and unlabeled data sets have no overlapping identities by construction. This is not realistic in large-scale face recognition, where one must contend with such overlaps, the frequency of which increases with the volume of data. Ignoring identity overlap leads to significant labeling noise, as data from the same identity is split into multiple clusters. To address this, we propose a novel identity separation method based on extreme value theory. It is formulated as an out-of-distribution detection algorithm, and greatly reduces the problems caused by overlapping-identity label noise. Considering cluster assignments as pseudo-labels, we must also overcome the labeling noise from clustering errors. We propose a modulation of the cosine loss, where the modulation weights correspond to an estimate of clustering uncertainty. Extensive experiments on both controlled and real settings demonstrate our method’s consistent improvements over supervised baselines, e.g., 11.6% improvement on IJB-A verification.

Publication Link: https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/4531_ECCV_2020_paper.php

Learning Monocular Visual Odometry via Self-Supervised Long-Term Modeling

August 23, 2020/in Publications/by NEC Labs America

Publication Date: 8/23/2020

Event: ECCV 2020 – The 16th European Conference on Computer Vision, Glasgow, UK

Reference: https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123590698.pdf

Authors: Yuliang Zou, Virginia Tech, NEC Laboratories America, Inc.; Pan Ji, NEC Laboratories America, Inc.; Quoc-Huy Tran, NEC Laboratories America, Inc.; Jia-Bin Huang, Virginia Tech; Manmohan Chandraker, NEC Laboratories America, Inc., UCSD

Abstract: Monocular visual odometry (VO) suffers severely from error accumulation during frame-to-frame pose estimation. In this paper, we present a self-supervised learning method for VO with special consideration for consistency over longer sequences. To this end, we model the long-term dependency in pose prediction using a pose network that features a two-layer convolutional LSTM module. We train the networks with purely self-supervised losses, including a cycle consistency loss that mimics the loop closure module in geometric VO. Inspired by prior geometric systems, we allow the networks to see beyond a small temporal window during training, through a novel a loss that incorporates temporally distant ( g $O(100)$) frames. Given GPU memory constraints, we propose a stage-wise training mechanism, where the first stage operates in a local time window and the second stage refines the poses with a “global” loss given the first stage features. We demonstrate competitive results on several standard VO datasets, including KITTI and TUM RGB-D.

Publication Link: https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/2256_ECCV_2020_paper.php

Learning to Optimize Domain Specific Normalization for Domain Generalization

August 23, 2020/in Publications/by NEC Labs America

Publication Date: 8/23/2020

Event: ECCV 2020 – The 16th European Conference on Computer Vision, Glasgow, UK

Reference: https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123670069.pdf

Authors: Seonguk Seo, Seoul National University; Yumin Suh, NEC Laboratories America, Inc.; Dongwan Kim, Seoul National University; Geeho Kim, Seoul National University; Jongwoo Han, LG Electronics; Bohyung Han, Seoul National University

Abstract: We propose a simple but effective multi-source domain generalization technique based on deep neural networks by incorporating optimized normalization layers that are specific to individual domains. Our approach employs multiple normalization methods while learning separate affine parameters per domain. For each domain, the activations are normalized by a weighted average of multiple normalization statistics. The normalization statistics are kept track of separately for each normalization type if necessary. Specifically, we employ batch and instance normalizations in our implementation to identify the best combination of these two normalization methods in each domain. The optimized normalization layers are effective to enhance the generalizability of the learned model. We demonstrate the state-of-the-art accuracy of our algorithm in the standard domain generalization benchmarks, as well as viability to further tasks such as multi-source domain adaptation and domain generalization in the presence of label noise.

Publication Link: https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/3935_ECCV_2020_paper.php

Object Detection with a Unified Label Space from Multiple Datasets

August 23, 2020/in Publications/by NEC Labs America

Publication Date: 8/23/2020

Event: ECCV 2020 – The 16th European Conference on Computer Vision, Glasgow, UK

Reference: https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123590171.pdf

Authors: Xiangyun Zhao, Northwestern University, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Gaurav Sharma, NEC Laboratories America, Inc.; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.; Ying Wu, Northwestern University

Abstract: Given multiple datasets with different label spaces, the goal of this work is to train a single object detector predicting over the union of all the label spaces. The practical benefits of such an object detector are obvious and significant—application-relevant categories can be picked and merged form arbitrary existing datasets. However, naive merging of datasets is not possible in this case, due to inconsistent object annotations. Consider an object category like faces that is annotated in one dataset, but is not annotated in another dataset, although the object itself appears in the later’s images. Some categories, like face here, would thus be considered foreground in one dataset, but background in another. To address this challenge, we design a framework which works with such partial annotations, and we exploit a pseudo labeling approach that we adapt for our specific case. We propose loss functions that carefully integrate partial but correct annotations with complementary but noisy pseudo labels. Evaluation in the proposed novel setting requires full annotation on the test set. We collect the required annotations and define a new challenging experimental setup for this task based on existing public datasets. We show improved performances compared to competitive baselines and appropriate adaptations of existing work

Publication Link: https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/2113_ECCV_2020_paper.php

Shuffle and Attend: Video Domain Adaptation

August 23, 2020/in Publications/by NEC Labs America

Publication Date: 8/23/2020

Event: ECCV 2020 – The 16th European Conference on Computer Vision, Glasgow, UK

Reference: https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123570664.pdf

Authors: Jinwoo Choi, Virginia Tech, NEC Laboratories America, Inc.; Gaurav Sharma, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Jia-Bin Huang, Virginia Tech

Abstract: We address the problem of domain adaptation in videos for the task of human action recognition. Inspired by image-based domain adaptation, we can perform video adaptation by aligning the features of frames or clips of source and target videos. However, equally aligning all clips is sub-optimal as not all clips are informative for the task. As the first novelty, we propose an attention mechanism which focuses on more discriminative clips and directly optimizes for video-level (cf. clip-level) alignment. As the backgrounds are often very different between source and target, the source background-corrupted model adapts poorly to target domain videos. To alleviate this, as a second novelty, we propose to use the clip order prediction as an auxiliary task. The clip order prediction loss, when combined with domain adversarial loss, encourages learning of representations which focus on the humans and objects involved in the actions, rather than the uninformative and widely differing (between source and target) backgrounds. We empirically show that both components contribute positively towards adaptation performance. We report state-of-the-art performances on two out of three challenging public benchmarks, two based on the UCF and HMDB datasets, and one on Kinetics to NEC-Drone datasets. We also support the intuitions and the results with qualitative results.

Publication Link: https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/1712_ECCV_2020_paper.php

SMART: Simultaneous Multi-Agent Recurrent Trajectory Prediction

August 23, 2020/in Publications/by NEC Labs America

Publication Date: 8/23/2020

Event: ECCV 2020 – The 16th European Conference on Computer Vision, Glasgow, UK

Reference: https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123720460.pdf

Authors: Sriram Narayanan, NEC Laboratories America, Inc.; Buyu Liu, NEC Laboratories America, Inc.; Francesco Pittaluga, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: We propose advances that address two key challenges in future trajectory prediction: (i) multimodality in both training data and predictions and (ii) constant time inference regardless of number of agents. Existing trajectory predictions are fundamentally limited by lack of diversity in training data, which is difficult to acquire with sufficient coverage of possible modes. Our first contribution is an automatic method to simulate diverse trajectories in the top-view. It uses pre-existing datasets and maps as initialization, mines existing trajectories to represent realistic driving behaviors and uses a multi-agent vehicle dynamics simulator to generate diverse new trajectories that cover various modes and are consistent with scene layout constraints. Our second contribution is a novel method that generates diverse predictions while accounting for scene semantics and multi-agent interactions, with constant-time inference independent of the number of agents. We propose a convLSTM with novel state pooling operations and losses to predict scene-consistent states of multiple agents in a single forward pass, along with a CVAE for diversity. We validate our proposed multi-agent trajectory prediction approach by training and testing on the proposed simulated dataset and existing real datasets of traffic scenes. In both cases, our approach outperforms SOTA methods by a large margin, highlighting the benefits of both our diverse dataset simulation and constant-time diverse trajectory prediction methods.”

Publication Link: https://www.ecva.net/papers/eccv_2020/papers_ECCV/html/5809_ECCV_2020_paper.php

Learning Monocular Visual Odometry via Self Supervised Long Term Modeling (arXiv)

July 21, 2020/in Publications/by NEC Labs America

Publication Date: 7/21/2020

Event: arXiv

Reference: https://arxiv.org/abs/2007.10983

Authors: Yuliang Zou, Virginia Tech, NEC Laboratories America, Inc., Pan Ji, NEC Laboratories America, Inc., Quoc Huy Tran, NEC Laboratories America, Inc., Jia Bin Huang, Virginia Tech, Manmohan Chandraker, NEC Laboratories America, Inc., UCSD

Abstract: Monocular visual odometry (VO) suffers severely from error accumulation during frame to frame pose estimation. In this paper, we present a self supervised learning method for VO with special consideration for consistency over longer sequences. To this end, we model the long term dependency in pose prediction using a pose network that features a two layer convolutional LSTM module. We train the networks with purely self supervised losses, including a cycle consistency loss that mimics the loop closure module in geometric VO. Inspired by prior geometric systems, we allow the networks to see beyond a small temporal window during training, through a novel a loss that incorporates temporally distant (e.g., O(100)) frames. Given GPU memory constraints, we propose a stage wise training mechanism, where the first stage operates in a local time window and the second stage refines the poses with a “global” loss given the first stage features. We demonstrate competitive results on several standard VO datasets, including KITTI and TUM RGB D.

Publication Link: https://arxiv.org/pdf/2007.10983.pdf

Improving Face Recognition by Clustering Unlabeled Faces in the Wild (arXiv)

July 10, 2020/in Publications/by NEC Labs America

Publication Date: 7/10/2020

Event: arXiv

Reference: https://arxiv.org/abs/2007.06995

Authors: Aruni RoyChowdhury, University of Massachusetts Amherst, NEC Laboratories America, Inc., Xiang Yu, NEC Laboratories America, Inc., Kihyuk Sohn, NEC Laboratories America, Inc., Eric Miller, University of Massachusetts Amherst., Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego

Abstract: While deep face recognition has benefited significantly from large scale labeled data, current research is focused on leveraging unlabeled data to further boost performance, reducing the cost of human annotation. Prior work has mostly been in controlled settings, where the labeled and unlabeled data sets have no overlapping identities by construction. This is not realistic in large scale face recognition, where one must contend with such overlaps, the frequency of which increases with the volume of data. Ignoring identity overlap leads to significant labeling noise, as data from the same identity is split into multiple clusters. To address this, we propose a novel identity separation method based on extreme value theory. It is formulated as an out of distribution detection algorithm, and greatly reduces the problems caused by overlapping identity label noise. Considering cluster assignments as pseudo labels, we must also overcome the labeling noise from clustering errors. We propose a modulation of the cosine loss, where the modulation weights correspond to an estimate of clustering uncertainty. Extensive experiments on both controlled and real settings demonstrate our method’s consistent improvements over supervised baselines, e.g., 11.6% improvement on IJB A verification.

Publication Link: https://arxiv.org/pdf/2007.06995.pdf

Peek-a-boo: Occlusion Reasoning in Indoor Scenes with Plane Representations

June 16, 2020/0 Comments/in Publications/by NEC Labs America

Publication Date: 6/16/2020

Event: CVPR 2020

Reference: pp 113-121, 2020

Authors: Ziyu Jiang, TAMU, NEC Laboratories America, Inc.; Buyu Liu, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Zhangyang Wang, TAMU; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: We address the challenging task of occlusion-aware indoor 3D scene understanding. We represent scenes by a set of planes, where each one is defined by its normal, offset and two masks outlining (i) the extent of the visible part and (ii) the full region that consists of both visible and occluded parts of the plane. We infer these planes from a single input image with a novel neural network architecture. It consists of a two-branch category-specific module that aims to predict layout and objects of the scene separately so that different types of planes can be handled better. We also introduce a novel loss function based on plane warping that can leverage multiple views at training time for improved occlusion-aware reasoning. In order to train and evaluate our occlusion-reasoning model, we use the ScanNet dataset and propose (i) a strategy to automatically extract ground truth for both visible and hidden regions and (ii) a new evaluation metric that specifically focuses on the prediction in hidden regions. We empirically demonstrate that our proposed approach can achieve higher accuracy for occlusion reasoning compared to competitive baselines on the ScanNet dataset, e.g. 42.65% relative improvement on hidden regions.

Publication Link: https://ieeexplore.ieee.org/document/9156416

Private-kNN Practical Differential Privacy for Computer Vision

June 16, 2020/in Publications/by NEC Labs America

Publication Date: 6/16/2020

Event: CVPR 2020

Reference: pp 11854-11862, 2020

Authors: Yuqing Zhu, University of California, Santa Barbara, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego; Yu-Xiang Wang, University of California, Santa Barbara

Abstract: With increasing ethical and legal concerns on privacy for deep models in visual recognition, differential privacy has emerged as a mechanism to disguise membership of sensitive data in training datasets. Recent methods like Private Aggregation of Teacher Ensembles (PATE) leverage a large ensemble of teacher models trained on disjoint subsets of private data, to transfer knowledge to a student model with privacy guarantees. However, labeled vision data is often expensive and datasets, when split into many disjoint training sets, lead to significantly sub-optimal accuracy and thus hardly sustain good privacy bounds. We propose a practically data-efficient scheme based on private release of k-nearest neighbor (kNN) queries, which altogether avoids splitting the training dataset. Our approach allows the use of privacy-amplification by subsampling and iterative refinement of the kNN feature embedding. We rigorously analyze the theoretical properties of our method and demonstrate strong experimental performance on practical computer vision datasets for face attribute recognition and person reidentification. In particular, we achieve comparable or better accuracy than PATE while reducing more than 90% of the privacy loss, thereby providing the “most practical method to-date” for private deep learning in computer vision.

Publication Link: https://ieeexplore.ieee.org/document/9156598

Towards Universal Representation Learning for Deep Face Recognition

June 16, 2020/in Publications/by NEC Labs America

Publication Date: 6/16/2020

Event: CVPR 2020

Reference: pp 6817-6826, 2020

Authors: Yichun Shi, Michigan State University, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego; Anil K. Jain, Michigan State University

Abstract: Recognizing wild faces is extremely hard as they appear with all kinds of variations. Traditional methods either train with specifically annotated variation data from target domains, or by introducing unlabeled target variation data to adapt from the training data. Instead, we propose a universal representation learning framework that can deal with larger variation unseen in the given training data without leveraging target domain knowledge. We firstly synthesize training data alongside some semantically meaningful variations, such as low resolution, occlusion and head pose. However, directly feeding the augmented data for training will not converge well as the newly introduced samples are mostly hard examples. We propose to split the feature embedding into multiple sub-embeddings, and associate different confidence values for each sub-embedding to smooth the training procedure. The sub-embeddings are further decorrelated by regularizing variation classification loss and variation adversarial loss on different partitions of them. Experiments show that our method achieves top performance on general face recognition datasets such as LFW and MegaFace, while significantly better on extreme benchmarks such as TinyFace and IJB-S.

Publication Link: https://ieeexplore.ieee.org/document/9156948

Understanding Road Layout from Videos as a Whole

June 16, 2020/0 Comments/in Publications/by NEC Labs America

Publication Date: 6/16/2020

Event: CVPR 2020

Reference: pp 4414-4423, 2020

Authors: Buyu Liu, NEC Laboratories America, Inc.; Bingbing Zhuang, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Pan Ji, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: In this paper, we address the problem of inferring the layout of complex road scenes from video sequences. To this end, we formulate it as a top-view road attributes prediction problem and our goal is to predict these attributes for each frame both accurately and consistently. In contrast to prior work, we exploit the following three novel aspects: leveraging camera motions in videos, including context cues and incorporating long-term video information. Specifically, we introduce a model that aims to enforce prediction consistency in videos. Our model consists of one LSTM and one Feature Transform Module (FTM). The former implicitly incorporates the consistency constraint with its hidden states, and the latter explicitly takes the camera motion into consideration when aggregating information along videos. Moreover, we propose to incorporate context information by introducing road participants, e.g. objects, into our model. When the entire video sequence is available, our model is also able to encode both local and global cues, e.g. information from both past and future frames. Experiments on two data sets show that: (1) Incorporating either global or contextual cues improves the prediction accuracy and leveraging both gives the best performance. (2) Introducing the LSTM and FTM modules improves the prediction consistency in videos. (3) The proposed method outperforms the SOTA by a large margin.

Publication Link: https://ieeexplore.ieee.org/document/9157339

Pseudo RGB D for Self Improving Monocular SLAM and Depth Prediction

April 22, 2020/in Publications/by NEC Labs America

Publication Date: 4/22/2020

Event: arXiv

Reference: https://arxiv.org/abs/2004.10681v1

Authors: Lokender Tiwari, NEC Laboratories America, Inc., Indraprastha Institute of Information Technology, Pan Ji, NEC Laboratories America, Inc., Quoc Huy Tran, NEC Laboratories America, Inc., Bingbing Zhuang, NEC Laboratories America, Inc., Saket Anand, Indraprastha Institute of Information Technology, Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: Classical monocular Simultaneous Localization And Mapping (SLAM) and the recently emerging convolutional neural networks (CNNs) for monocular depth prediction represent two largely disjoint approaches towards building a 3D map of the surrounding environment. In this paper, we demonstrate that the coupling of these two by leveraging the strengths of each mitigates the other’s shortcomings. Specifically, we propose a joint narrow and wide baseline based self improving framework, where on the one hand the CNN predicted depth is leveraged to perform pseudo RGB D feature based SLAM, leading to better accuracy and robustness than the monocular RGB SLAM baseline. On the other hand, the bundle adjusted 3D scene structures and camera poses from the more principled geometric SLAM are injected back into the depth network through novel wide baseline losses proposed for improving the depth prediction network, which then continues to contribute towards better pose and 3D structure estimation in the next iteration. We emphasize that our framework only requires unlabeled monocular videos in both training and inference stages, and yet is able to outperform state of the art self supervised monocular and stereo depth prediction networks (e.g, Monodepth2) and feature based monocular SLAM system (i.e, ORB SLAM). Extensive experiments on KITTI and TUM RGB D datasets verify the superiority of our self improving geometry CNN framework.

Publication Link: https://arxiv.org/pdf/2004.10681v1.pdf

SMART: Simultaneous Multi Agent Recurrent Trajectory prediction (arXiv)

March 6, 2020/in Publications/by NEC Labs America

Publication Date: 3/6/2020

Event: arXiv

Reference: https://arxiv.org/abs/2007.13078

Authors: Sriram Narayanan, NEC Laboratories America, Inc., Buyu Liu, NEC Laboratories America, Inc., Francesco Pittaluga, NEC Laboratories America, Inc., Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: We propose advances that address two key challenges in future trajectory prediction: (i) multimodality in both training data and predictions and (ii) constant time inference regardless of number of agents. Existing trajectory predictions are fundamentally limited by lack of diversity in training data, which is difficult to acquire with sufficient coverage of possible modes. Our first contribution is an automatic method to simulate diverse trajectories in the top view. It uses pre existing datasets and maps as initialization, mines existing trajectories to represent realistic driving behaviors and uses a multi agent vehicle dynamics simulator to generate diverse new trajectories that cover various modes and are consistent with scene layout constraints. Our second contribution is a novel method that generates diverse predictions while accounting for scene semantics and multi agent interactions, with constant time inference independent of the number of agents. We propose a convLSTM with novel state pooling operations and losses to predict scene consistent states of multiple agents in a single forward pass, along with a CVAE for diversity. We validate our proposed multi agent trajectory prediction approach by training and testing on the proposed simulated dataset and existing real datasets of traffic scenes. In both cases, our approach outperforms SOTA methods by a large margin, highlighting the benefits of both our diverse dataset simulation and constant time diverse trajectory prediction methods.

Publication Link: https://arxiv.org/pdf/2007.13078.pdf

Active Adversarial Domain Adaptation

March 2, 2020/in Publications/by NEC Labs America

Publication Date: 3/2/2020

Event: WACV 2020, Snowmass Village, CO USA

Reference: pp 728-737, 2020

Authors: Jong-Chyi Su, University of Massachusetts, Amherst, NEC Laboratories America, Inc.; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Buyu Liu, NEC Laboratories America, Inc.; Subhransu Maji, University of Massachusetts, Amherst; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: We propose an active learning approach for transferring representations across domains. Our approach, active adversarial domain adaptation (AADA), explores a duality between two related problems: adversarial domain alignment and importance sampling for adapting models across domains. The former uses a domain discriminative model to align domains, while the latter utilizes the model to weigh samples to account for distribution shifts. Specifically, our importance weight promotes unlabeled samples with large uncertainty in classification and diversity compared to la-beled examples, thus serving as a sample selection scheme for active learning. We show that these two views can be unified in one framework for domain adaptation and transfer learning when the source domain has many labeled examples while the target domain does not. AADA provides significant improvements over fine-tuning based approaches and other sampling methods when the two domains are closely related. Results on challenging domain adaptation tasks such as object detection demonstrate that the advantage over baseline approaches is retained even after hundreds of examples being actively annotated.

Publication Link: https://ieeexplore.ieee.org/document/9093390

Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zero-shot Classification and Retrieval of Videos

March 2, 2020/in Publications/by NEC Labs America

Publication Date: 3/2/2020

Event: WACV 2020, Snowmass Village, CO USA

Reference: pp 3240-3249, 2020

Authors: Kranti Kumar Parida, IIT Kanpur; Neeraj Matiyali, IIT Kanpur; Tanaya Guha, University of Warwick; Gaurav Sharma, NEC Laboratories America, Inc.

Abstract: We present an audio-visual multimodal approach for the task of zero-shot learning (ZSL) for classification and retrieval of videos. ZSL has been studied extensively in the recent past but has primarily been limited to visual modality and to images. We demonstrate that both audio and visual modalities are important for ZSL for videos. Since a dataset to study the task is currently not available, we also construct an appropriate multimodal dataset with 33 classes containing 156, 416 videos, from an existing large scale audio event dataset. We empirically show that the performance improves by adding audio modality for both tasks of zero-shot classification and retrieval, when using multi-modal extensions of embedding learning methods. We also propose a novel method to predict the `dominant’ modality using a jointly learned modality attention network. We learn the attention in a semi-supervised setting and thus do not require any additional explicit labelling for the modalities. We provide qualitative validation of the modality specific attention, which also successfully generalizes to unseen test classes.

Publication Link: https://ieeexplore.ieee.org/document/9093438

DAVID: Dual-Attentional Video Deblurring

March 2, 2020/in Publications/by NEC Labs America

Publication Date: 3/2/2020

Event: WACV 2020, Snowmass Village, CO USA

Reference: pp 2365-2374, 2020

Authors: Junru Wu, Texas Agriculture Mechanics University, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego; Zhangyang Wang, Texas Agriculture Mechanics University; Ding Liu, ByteDance AI Lab

Abstract: Blind video deblurring restores sharp frames from a blurry sequence without any prior. It is a challenging task because the blur due to camera shake, object movement and defocusing is heterogeneous in both temporal and spatial dimensions. Traditional methods train on datasets synthesized with a single level of blur, and thus do not generalize well across levels of blurriness. To address this challenge, we propose a dual attention mechanism to dynamically aggregate temporal cues for deblurring with an end-to-end trainable network structure. Specifically, an internal attention module adaptively selects the optimal temporal scales for restoring the sharp center frame. An external attention module adaptively aggregates and refines multiple sharp frame estimates, from several internal attention modules designed for different blur levels. To train and evaluate on more diverse blur severity levels, we propose a Challenging DVD dataset generated from the raw DVD video set by pooling frames with different temporal windows. Our framework achieves consistently better performance on this more challenging dataset while obtaining strongly competitive results on the original DVD benchmark. Extensive ablative studies and qualitative visualizations further demonstrate the advantage of our method in handling real video blur.

Publication Link: https://ieeexplore.ieee.org/document/9093529

Unsupervised and Semi-Supervised Domain Adaptation for Action Recognition from Drones

March 2, 2020/in Publications/by NEC Labs America

Publication Date: 3/2/2020

Event: WACV 2020, Snowmass Village, CO USA

Reference: pp 1706-1715, 2020

Authors: Jinwoo Choi, Virginia Tech, NEC Laboratories America, Inc.; Gaurav Sharma, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.; Jia-Bin Huang, Virginia Tech

Abstract: We address the problem of human action classification in drone videos. Due to the high cost of capturing and labeling large-scale drone videos with diverse actions, we present unsupervised and semi-supervised domain adaptation approaches that leverage both the existing fully annotated action recognition datasets and unannotated (or only a few annotated) videos from drones. To study the emerging problem of drone-based action recognition, we create a new dataset, NEC-DRONE, containing 5,250 videos to evaluate the task. We tackle both problem settings with 1) same and 2) different action label sets for the source (e.g., Kinectics dataset) and target domains (drone videos). We present a combination of video and instance-based adaptation methods, paired with either a classifier or an embedding-based framework to transfer the knowledge from source to target. Our results show that the proposed adaptation approach substantially improves the performance on these challenging and practical tasks. We further demonstrate the applicability of our method for learning cross-view action recognition on the Charades-Ego dataset. We provide qualitative analysis to understand the behaviors of our approaches.

Publication Link: https://ieeexplore.ieee.org/document/9093511

Video Person Re-Identification using Learned Clip Similarity Aggregation

March 2, 2020/in Publications/by NEC Labs America

Publication Date: 3/2/2020

Event: WACV 2020, Snowmass Village, CO USA

Reference: pp 2644-2653, 2020

Authors: Neeraj Matiyali, IIT Kanpur; Gaurav Sharma, NEC Laboratories America, Inc.

Abstract: We address the challenging task of video-based person re-identification. Recent works have shown that splitting the video sequences into clips and then aggregating clip-based similarity is appropriate for the task. We show that using a learned clip similarity aggregation function allows filtering out hard clip pairs, e.g. where the person is not clearly visible, is in a challenging pose, or where the poses in the two clips are too different to be informative. This allows the method to focus on clip-pairs which are more informative for the task. We also introduce the use of 3D CNNs for video-based re-identification and show their effectiveness by performing equivalent to previous works, which use optical flow in addition to RGB, while using RGB inputs only. We give quantitative results on three challenging public benchmarks and show better or competitive performance. We also validate our method qualitatively.

Publication Link: https://ieeexplore.ieee.org/document/9093510

Adversarial Learning of Privacy-Preserving and Task-Oriented Representations

February 7, 2020/in Publications/by NEC Labs America

Publication Date: 2/7/2020

Event: AAAI 2020, New York, New York USA

Reference: pp 12434-12441, 2020

Authors: Taihong Xiao, University of California at Merced; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc., Google; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Ming-Hsuan Yang, University of California at Merced

Abstract: Data privacy has emerged as an important issue as data-driven deep learning has been an essential component of modern machine learning systems. For instance, there could be a potential privacy risk of machine learning systems via the model inversion attack, whose goal is to reconstruct the input data from the latent representation of deep networks. Our work aims at learning a privacy-preserving and task-oriented representation to defend against such model inversion attacks. Specifically, we propose an adversarial reconstruction learning framework that prevents the latent representations decoded into original input data. By simulating the expected behavior of adversary, our framework is realized by minimizing the negative pixel reconstruction loss or the negative feature reconstruction (i.e., perceptual distance) loss. We validate the proposed method on face attribute prediction, showing that our method allows protecting visual privacy with a small decrease in utility performance. In addition, we show the utility-privacy trade-off with different choices of hyperparameter for negative perceptual distance loss at training, allowing service providers to determine the right level of privacy-protection with a certain utility performance. Moreover, we provide an extensive study with different selections of features, tasks, and the data to further analyze their influence on privacy protection.

Publication Link: https://aaai.org/papers/12434-adversarial-learning-of-privacy-preserving-and-task-oriented-representations/

DAVID: Dual-Attentional Video Deblurring (arXiv)

December 7, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 12/7/2019

Event: arXiv

Reference: https://arxiv.org/abs/1912.03445v1

Authors: Junru Wu, Texas Agriculture Mechanics University, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Ding Liu, ByteDance AI Lab; Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego; Zhangyang Wang, Texas Agriculture Mechanics University

Abstract: Blind video deblurring restores sharp frames from a blurry sequence without any prior. It is a challenging task because the blur due to camera shake, object movement and defocusing is heterogeneous in both temporal and spatial dimensions. Traditional methods train on datasets synthesized with a single level of blur, and thus do not generalize well across levels of blurriness. To address this challenge, we propose a dual attention mechanism to dynamically aggregate temporal cues for deblurring with an end-to-end trainable network structure. Specifically, an internal attention module adaptively selects the optimal temporal scales for restoring the sharp center frame. An external attention module adaptively aggregates and refines multiple sharp frame estimates, from several internal attention modules designed for different blur levels. To train and evaluate on more diverse blur severity levels, we propose a Challenging DVD dataset generated from the raw DVD video set by pooling frames with different temporal windows. Our framework achieves consistently better performance on this more challenging dataset while obtaining strongly competitive results on the original DVD benchmark. Extensive ablative studies and qualitative visualizations further demonstrate the advantage of our method in handling real video blur.

Publication Link: https://arxiv.org/pdf/1912.03445v1.pdf

Degeneracy in Self-Calibration Revisited and a Deep Learning Solution for Uncalibrated SLAM

November 3, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 11/3/2019

Event: IROS 2019, The Venetian Macao, Macau, China

Reference: pp 3766-3773, 2019

Authors: Bingbing Zhuang, National University of Singapore, NEC Laboratories America, Inc.; Quoc-Huy Tran, NEC Laboratories America, Inc.; Loong Fah Cheong, National University of Singapore; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Self-calibration of camera intrinsics and radial distortion has a long history of research in the computer vision community. However, it remains rare to see real applications of such techniques to modern Simultaneous Localization And Mapping (SLAM) systems, especially in driving scenarios. In this paper, we revisit the geometric approach to this problem, and provide a theoretical proof that explicitly shows the ambiguity between radial distortion and scene depth when two-view geometry is used to self-calibrate the radial distortion. In view of such geometric degeneracy, we propose a learning approach that trains a convolutional neural network (CNN) on a large amount of synthetic data. We demonstrate the utility of our proposed method by applying it as a checkerboard-free calibration tool for SLAM, achieving comparable or superior performance to previous learning and hand-crafted method

Publication Link: https://ieeexplore.ieee.org/document/8967912

Learning 2D to 3D Lifting for Object Detection in 3D for Autonomous Vehicles

November 3, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 11/3/2019

Event: IROS 2019, The Venetian Macao, Macau, China

Reference: pp 4504-4511, 2019

Authors: Siddharth Srivastava , IIT Delhi; Frederic Jurie , University of Caen Normadie; Gaurav Sharma , NEC Laboratories America, Inc.

Abstract: We address the problem of 3D object detection from 2D monocular images in autonomous driving scenarios. We propose to lift the 2D images to 3D representations using learned neural networks and leverage existing networks working directly on 3D data to perform 3D object detection and localization. We show that, with carefully designed training mechanism and automatically selected minimally noisy data, such a method is not only feasible, but gives higher results than many methods working on actual 3D inputs acquired from physical sensors. On the challenging KITTI benchmark, we show that our 2D to 3D lifted method outperforms many recent competitive 3D networks while significantly outperforming previous state-of-the-art for 3D detection from monocular images. We also show that a late fusion of the output of the network trained on generated 3D images, with that trained on real 3D images, improves performance. We find the results very interesting and argue that such a method could serve as a highly reliable backup in case of malfunction of expensive 3D sensors, if not potentially making them redundant, at least in the case of low human injury risk autonomous navigation scenarios like warehouse automation.

Publication Link: https://ieeexplore.ieee.org/document/8967624

Domain Adaptation for Structured Output via Discriminative Patch Representations

October 27, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/27/2019

Event: ICCV 2019 – International Conference on Computer Vision 2019, Seoul, Korea

Reference: pp 1456-1465

Authors: Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: Predicting structured outputs such as semantic segmentation relies on expensive per-pixel annotations to learn supervised models like convolutional neural networks. However, models trained on one data domain may not generalize well to other domains without annotations for model finetuning. To avoid the labor-intensive process of annotation, we develop a domain adaptation method to adapt the source data to the unlabeled target domain. We propose to learn discriminative feature representations of patches in the source domain by discovering multiple modes of patch-wise output distribution through the construction of a clustered space. With such representations as guidance, we use an adversarial learning scheme to push the feature representations of target patches in the clustered space closer to the distributions of source patches. In addition, we show that our framework is complementary to existing domain adaptation techniques and achieves consistent improvements on semantic segmentation. Extensive ablations and results are demonstrated on numerous benchmark datasets with various settings, such as synthetic-to-real and cross-city scenarios.

Publication Link: https://openaccess.thecvf.com/content_ICCV_2019/html/Tsai_Domain_Adaptation_for_Structured_Output_via_Discriminative_Patch_Representations_ICCV_2019_paper.html

GLoSH: Global-Local Spherical Harmonics for Intrinsic Image Decomposition

October 27, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/27/2019

Event: ICCV 2019 – International Conference on Computer Vision 2019, Seoul, Korea

Reference: pp 7819-7828, 2019

Authors: Hao Zhou, University of Maryland; Xiang Yu, NEC Laboratories America, Inc.; David Jacobs, University of Maryland

Abstract: Traditional intrinsic image decomposition focuses on decomposing images into reflectance and shading, leaving surfaces normals and lighting entangled in shading. In this work, we propose a Global-Local Spherical Harmonics (GLoSH) lighting model to improve the lighting component, and jointly predict reflectance and surface normals. The global SH models the holistic lighting while local SH account for the spatial variation of lighting. Also, a novel non-negative lighting constraint is proposed to encourage the estimated SH to be physically meaningful. To seamlessly reflect the GLoSH model, we design a coarse-to-fine network structure. The coarse network predicts global SH, reflectance and normals, and the fine network predicts their local residuals. Lacking labels for reflectance and lighting, we apply synthetic data for model pre-training and fine-tune the model with real data in a self-supervised way. Compared to the state-of-the-art methods only targeting normals or reflectance and shading, our method recovers all components and achieves consistently better results on three real datasets, IIW, SAW and NYUv2.

Publication Link: https://openaccess.thecvf.com/content_ICCV_2019/html/Zhou_GLoSH_Global-Local_Spherical_Harmonics_for_Intrinsic_Image_Decomposition_ICCV_2019_paper.html

Video Person Re-Identification using Learned Clip Similarity Aggregation (arXiv)

October 14, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/14/2019

Event: arXiv

Reference: https://arxiv.org/abs/1910.08055

Authors: Neeraj Matiyali, IIT Kanpur; Gaurav Sharma, NEC Laboratories America, Inc.

Abstract: We address the challenging task of video-based person re-identification. Recent works have shown that splitting the video sequences into clips and then aggregating clip based similarity is appropriate for the task. We show that using a learned clip similarity aggregation function allows filtering out hard clip pairs, e.g. where the person is not clearly visible, is in a challenging pose, or where the poses in the two clips are too different to be informative. This allows the method to focus on clip-pairs which are more informative for the task. We also introduce the use of 3D CNNs for video-based re-identification and show their effectiveness by performing equivalent to previous works, which use optical flow in addition to RGB, while using RGB inputs only. We give quantitative results on three challenging public benchmarks and show better or competitive performance. We also validate our method qualitatively.

Publication Link: https://arxiv.org/pdf/1910.08055.pdf

Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videos

October 11, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/11/2019

Event: arXiv

Reference: https://arxiv.org/abs/1910.08732

Authors: Kranti Kumar Parida, IIT Kanpur; Neeraj Matiyali, IIT Kanpur; Tanaya Guha, University of Warwick; Gaurav Sharma, NEC Laboratories America, Inc.

Abstract: We present an audio-visual multimodal approach for the task of zeroshot learning (ZSL) for classification and retrieval of videos. ZSL has been studied extensively in the recent past but has primarily been limited to visual modality and to images. We demonstrate that both audio and visual modalities are important for ZSL for videos. Since a dataset to study the task is currently not available, we also construct an appropriate multimodal dataset with 33 classes containing 156,416 videos, from an existing large scale audio event dataset. We empirically show that the performance improves by adding audio modality for both tasks of zeroshot classification and retrieval, when using multimodal extensions of embedding learning methods. We also propose a novel method to predict the `dominant’ modality using a jointly learned modality attention network. We learn the attention in a semi-supervised setting and thus do not require any additional explicit labelling for the modalities. We provide qualitative validation of the modality specific attention, which also successfully generalizes to unseen test classes.

Publication Link: https://arxiv.org/pdf/1910.08732.pdf

Deep Supervision with Intermediate Concepts (IEEE)

August 1, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 8/1/2019

Event: IEEE Transactions on Pattern Analysis and Machine Intelligence

Reference: Vol. 41, No. 8, pp 1828-1843, 2019

Authors: Chi Li, Johns Hopkins University, NEC Laboratories America, Inc.; M. Zeeshan Zia, NEC Laboratories America, Inc.; Quoc-Huy Tran, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Gregory D. Hager, Johns Hopkins University; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Recent data-driven approaches to scene interpretation predominantly pose inference as an end-to-end black-box mapping, commonly performed by a Convolutional Neural Network (CNN). However, decades of work on perceptual organization in both human and machine vision suggest that there are often intermediate representations that are intrinsic to an inference task, and which provide essential structure to improve generalization. In this work, we explore an approach for injecting prior domain structure into neural network training by supervising hidden layers of a CNN with intermediate concepts that normally are not observed in practice. We formulate a probabilistic framework which formalizes these notions and predicts improved generalization via this deep supervision method. One advantage of this approach is that we are able to train only from synthetic CAD renderings of cluttered scenes, where concept values can be extracted, but apply the results to real images. Our implementation achieves the state-of-the-art performance of 2D/3D keypoint localization and image classification on real image benchmarks including KITTI, PASCALVOC, PASCAL3D+, IKEA, and CIFAR100. We provide additional evidence that our approach outperforms alternative forms of supervision, such as multi-task networks.

Publication Link: https://ieeexplore.ieee.org/document/8434117

Degeneracy in Self-Calibration Revisited and a Deep Learning Solution for Uncalibrated SLAM (arXiv)

July 29, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 7/29/2019

Event: arXiv

Reference: https://arxiv.org/abs/1907.13185

Authors: Bingbing Zhuang, National University of Singapore, NEC Laboratories America, Inc.; Quoc-Huy Tran, NEC Laboratories America, Inc.; Pan Ji, NEC Laboratories America, Inc.; Gim Hee Lee, National University of Singapore; Loong Fah Cheong, National University of Singapore; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Self-calibration of camera intrinsics and radial distortion has a long history of research in the computer vision community. However, it remains rare to see real applications of such techniques to modern Simultaneous Localization And Mapping (SLAM) systems, especially in driving scenarios. In this paper, we revisit the geometric approach to this problem, and provide a theoretical proof that explicitly shows the ambiguity between radial distortion and scene depth when two-view geometry is used to self-calibrate the radial distortion. In view of such geometric degeneracy, we propose a learning approach that trains a convolutional neural network (CNN) on a large amount of synthetic data. We demonstrate the utility of our proposed method by applying it as a checkerboard-free calibration tool for SLAM, achieving comparable or superior performance to previous learning and hand-crafted methods.

Publication Link: https://arxiv.org/pdf/1907.13185.pdf

Pose-variant 3D Facial Attribute Generation

July 23, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 7/23/2019

Event: arXiv

Reference: https://arxiv.org/abs/1907.10202

Authors: Feng-Ju Chang, University of Southern California, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Ram Nevatia, University of Southern California; Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego

Abstract: We address the challenging problem of generating facial attributes using a single image in an unconstrained pose. In contrast to prior works that largely consider generation on 2D near-frontal images, we propose a GAN-based framework to generate attributes directly on a dense 3D representation given by UV texture and position maps, resulting in photorealistic, geometrically-consistent and identity-preserving outputs. Starting from a self-occluded UV texture map obtained by applying an off-the-shelf 3D reconstruction method, we propose two novel components. First, a texture completion generative adversarial network (TC-GAN) completes the partial UV texture map. Second, a 3D attribute generation GAN (3DA-GAN) synthesizes the target attribute while obtaining an appearance consistent with 3D face geometry and preserving identity. Extensive experiments on CelebA, LFW and IJB-A show that our method achieves consistently better attribute generation accuracy than prior methods, a higher degree of qualitative photorealism and preserves face identity information.

Publication Link: https://arxiv.org/pdf/1907.10202.pdf

A Dataset for High-level 3D Scene Understanding of Complex Road Scenes in the Top-View

June 17, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 6/17/2019

Event: Proceedings of CVPR 2019 Workshop on 3D Scene Understanding for Vision, Graphics, and Robotics

Reference: pp. 1-4, 2019

Authors: Ziyan Wang, Carnegie Mellon University, NEC Laboratories America, Inc.; Buyu Liu, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: We introduce a novel dataset for high-level 3D scene understanding of complex road scenes. Our annotations extend the existing datasets KITTI [5] and nuScenes [1] with semantically and geometrically meaningful attributes like the number of lanes or the existence of, and distance to, intersections, sidewalks and crosswalks. Our attributes are rich enough to build a meaningful representation of the scene in the top-view and provide a tangible interface to the real world for several practical applications

Publication Link: https://scene-understanding.com/2019/program.html

A Parametric Top-View Representation of Complex Road Scenes

June 16, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 6/16/2019

Event: IEEE Computer Vision and Pattern Recognition (CVPR 2019)

Reference: pp. 10325-10333

Authors: Ziyan Wang , Carnegie Mellon University, NEC Laboratories America, Inc.; Buyu Liu, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: In this paper, we address the problem of inferring the layout of complex road scenes given a single camera as input. To achieve that, we first propose a novel parameterized model of road layouts in a top-view representation, which is not only intuitive for human visualization but also provides an interpretable interface for higher-level decision making. Moreover, the design of our top-view scene model allows for efficient sampling and thus generation of large-scale simulated data, which we leverage to train a deep neural network to infer our scene model’s parameters. Specifically, our proposed training procedure uses supervised domain-adaptation techniques to incorporate both simulated as well as manually annotated data. Finally, we design a Conditional Random Field (CRF) that enforces coherent predictions for a single frame and encourages temporal smoothness among video frames. Experiments on two public data sets show that: (1) Our parametric top-view model is representative enough to describe complex road scenes; (2) The proposed method outperforms baselines trained on manually-annotated or simulated data only, thus getting the best of both; (3) Our CRF is able to generate temporally smoothed while semantically meaningful results.

Publication Link: https://openaccess.thecvf.com/content_CVPR_2019/html/Wang_A_Parametric_Top-View_Representation_of_Complex_Road_Scenes_CVPR_2019_paper.html

Feature Transfer Learning for Face Recognition with Under-Represented Data

June 16, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 6/16/2019

Event: IEEE Computer Vision and Pattern Recognition (CVPR 2019)

Reference: pp. 5704-5713, 2019

Authors: Xi Yin, Michigan State University, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Xiaoming Liu, Michigan State University; Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego

Abstract: Despite the large volume of face recognition datasets, there is a significant portion of subjects, of which the samples are insufficient and thus under-represented. Ignoring such significant portion results in insufficient training data. Training with under-represented data leads to biased classifiers in conventionally-trained deep networks. In this paper, we propose a center-based feature transfer framework to augment the feature space of under-represented subjects from the regular subjects that have sufficiently diverse samples. A Gaussian prior of the variance is assumed across all subjects and the variance from regular ones are transferred to the under-represented ones. This encourages the under-represented distribution to be closer to the regular distribution. Further, an alternating training regimen is proposed to simultaneously achieve less biased classifiers and a more discriminative feature representation. We conduct ablative study to mimic the under-represented datasets by varying the portion of under-represented classes on the MS-Celeb-1M dataset. Advantageous results on LFW, IJB-A and MS-Celeb-1M demonstrate the effectiveness of our feature transfer and training strategy, compared to both general baselines and state-of-the-art methods. Moreover, our feature transfer successfully presents smooth visual interpolation, which conducts disentanglement to preserve identity of a class while augmenting its feature space with non-identity variations such as pose and lighting.

Publication Link: https://openaccess.thecvf.com/content_CVPR_2019/html/Yin_Feature_Transfer_Learning_for_Face_Recognition_With_Under-Represented_Data_CVPR_2019_paper.html

Gotta Adapt Em All: Joint Pixel and Feature-Level Domain Adaptation for Recognition in the Wild

June 16, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 6/16/2019

Event: IEEE Computer Vision and Pattern Recognition (CVPR 2019)

Reference: pp. 2672-2681, 2019

Authors: Luan Tran, Michigan State University, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Xiaoming Liu, Michigan State University; Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego

Abstract: Recent developments in deep domain adaptation have allowed knowledge transfer from a labeled source domain to an unlabeled target domain at the level of intermediate features or input pixels. We propose that advantages may be derived by combining them, in the form of different insights that lead to a novel design and complementary properties that result in better performance. At the feature level, inspired by insights from semi-supervised learning, we propose a classification-aware domain adversarial neural network that brings target examples into more classifiable regions of source domain. Next, we posit that computer vision insights are more amenable to injection at the pixel level. In particular, we use 3D geometry and image synthesis based on a generalized appearance flow to preserve identity across pose transformations, while using an attribute-conditioned CycleGAN to translate a single source into multiple target images that differ in lower-level properties such as lighting. Besides standard UDA benchmark, we validate on a novel and apt problem of car recognition in unlabeled surveillance images using labeled images from the web, handling explicitly specified, nameable factors of variation through pixel-level and implicit, unspecified factors through feature-level adaptation.

Publication Link: https://openaccess.thecvf.com/content_CVPR_2019/html/Tran_Gotta_Adapt_Em_All_Joint_Pixel_and_Feature-Level_Domain_Adaptation_CVPR_2019_paper.html

Learning Structure-And-Motion-Aware Rolling Shutter Correction

June 16, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 6/16/2019

Event: IEEE Computer Vision and Pattern Recognition (CVPR 2019)

Reference: pp. 4551-4560, 2019

Authors: Bingbing Zhuang, National University of Singapore, NEC Laboratories America, Inc.; Quoc-Huy Tran, NEC Laboratories America, Inc.; Pan Ji, NEC Laboratories America, Inc.; Loong Fah Cheong, National University of Singapore; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: An exact method of correcting the rolling shutter (RS) effect requires recovering the underlying geometry, i.e. the scene structures and the camera motions between scanlines or between views. However, the multiple-view geometry for RS cameras is much more complicated than its global shutter (GS) counterpart, with various degeneracies. In this paper, we first make a theoretical contribution by showing that RS two-view geometry is degenerate in the case of pure translational camera motion. In view of the complex RS geometry, we then propose a Convolutional Neural Network (CNN)-based method which learns the underlying geometry (camera motion and scene structure) from just a single RS image and perform RS image correction. We call our method structure-and-motion-aware RS correction because it reasons about the concealed motions between the scanlines as well as the scene structure. Our method learns from a large-scale dataset synthesized in a geometrically meaningful way where the RS effect is generated in a manner consistent with the camera motion and scene structure. In extensive experiments, our method achieves superior performance compared to other state-of-the-art methods for single image RS correction and subsequent Structure from Motion (SfM) applications.

Publication Link: https://openaccess.thecvf.com/content_CVPR_2019/html/Zhuang_Learning_Structure-And-Motion-Aware_Rolling_Shutter_Correction_CVPR_2019_paper.html

Feature Transfer Learning for Face Recognition with Under-Represented Data (IEEE)

June 16, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 6/16/2019

Event: IEEE Computer Vision and Pattern Recognition (CVPR 2019)

Reference: pp. 5704-5713, 2019

Authors: Xi Yin, Michigan State University, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Xiaoming Liu, Michigan State University; Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego

Abstract: Despite the large volume of face recognition datasets, there is a significant portion of subjects, of which the samples are insufficient and thus under-represented. Ignoring such significant portion results in insufficient training data. Training with under-represented data leads to biased classifiers in conventionally-trained deep networks. In this paper, we propose a center-based feature transfer framework to augment the feature space of under-represented subjects from the regular subjects that have sufficiently diverse samples. A Gaussian prior of the variance is assumed across all subjects and the variance from regular ones are transferred to the under-represented ones. This encourages the under-represented distribution to be closer to the regular distribution. Further, an alternating training regimen is proposed to simultaneously achieve less biased classifiers and a more discriminative feature representation. We conduct ablative study to mimic the under-represented datasets by varying the portion of under-represented classes on the MS-Celeb-1M dataset. Advantageous results on LFW, IJB-A and MS-Celeb-1M demonstrate the effectiveness of our feature transfer and training strategy, compared to both general baselines and state-of-the-art methods. Moreover, our feature transfer successfully presents smooth visual interpolation, which conducts disentanglement to preserve identity of a class while augmenting its feature space with non-identity variations such as pose and lighting.

Publication Link: https://openaccess.thecvf.com/content_CVPR_2019/html/Yin_Feature_Transfer_Learning_for_Face_Recognition_With_Under-Represented_Data_CVPR_2019_paper.html

Unsupervised Domain Adaptation for Distance Metric Learning (CVPR)

June 16, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 6/16/2019

Event: Proceedings of CVPR 2019 Workshop on 3D Scene Understanding for Vision, Graphics, and Robotics

Reference: pp. 1-18, 2019

Authors: Kihyuk Sohn, NEC Laboratories America, Inc.; Wenling Shang, University of Amsterdam; Xiang Yu, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Unsupervised domain adaptation is a promising avenue to enhance the performance of deep neural networks on a target domain, using labels only from a source domain. However, the two predominant methods, domain discrepancy reduction learning and semi-supervised learning, are not readily applicable when source and target domains do not share a common label space. This paper addresses the above scenario by learning a representation space that retains discriminative power on both the (labeled) source and (unlabeled) target domains while keeping representations for the two domains well-separated. Inspired by a theoretical analysis, we first reformulate the disjoint classification task, where the source and target domains correspond to non-overlapping class labels, to a verification one. To handle both within and cross domain verifications, we propose a Feature Transfer Network (FTN) to separate the target feature space from the original source space while aligned with a transformed source space. Moreover, we present a non-parametric multi-class entropy minimization loss to further boost the discriminative power of FTNs on the target domain. In experiments, we first illustrate how FTN works in a controlled setting of adapting from MNIST-M to MNIST with disjoint digit classes between the two domains and then demonstrate the effectiveness of FTNs throughstate-of-the-art performances on a cross-ethnicity face recognition problem.

Publication Link:

Neural Collaborative Subspace Clustering

June 9, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 6/9/2019

Event: International Conference on Machine Learning, ICML 2019, Long Beach, CA USA

Reference: pp 1-10, 2019

Authors: Tong Zhang, Australian National University; Pan Ji, NEC Laboratories America, Inc.; Mehrtash Harandi, University of Monash; Wenbing Huang, Tencent AI Lab; Hongdong Li, Australian National University

Abstract: We introduce the Neural Collaborative Subspace Clustering, a neural model that discovers clusters of data points drawn from a union of low-dimensional subspaces. In contrast to previous attempts, our model runs without the aid of spectral clustering. This makes our algorithm one of the kinds that can gracefully scale to large datasets. At its heart, our neural model benefits from a classifier which determines whether a pair of points lies on the same subspace or not. Essential to our model is the construction of two affinity matrices, one from the classifier and the other from a notion of subspace self-expressiveness, to supervise training in a collaborative scheme. We thoroughly assess and contrast the performance of our model against various state-of-the-art clustering algorithms including deep subspace-based ones.

Publication Link: https://proceedings.mlr.press/v97/zhang19g.html

Unsupervised Domain Adaptation for Distance Metric Learning

May 6, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 5/6/2019

Event: Seventh International Conference on Learning Representations (ICLR 2019)

Reference: pp. 1-18, 2019

Authors: Kihyuk Sohn, NEC Laboratories America, Inc.; Wenling Shang, University of Amsterdam; Xiang Yu, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Unsupervised domain adaptation is a promising avenue to enhance the performance of deep neural networks on a target domain, using labels only from a source domain. However, the two predominant methods, domain discrepancy reduction learning and semi-supervised learning, are not readily applicable when source and target domains do not share a common label space. This paper addresses the above scenario by learning a representation space that retains discriminative power on both the (labeled) source and (unlabeled) target domains while keeping representations for the two domains well-separated. Inspired by a theoretical analysis, we first reformulate the disjoint classification task, where the source and target domains correspond to non-overlapping class labels, to a verification one. To handle both within and cross domain verifications, we propose a Feature Transfer Network (FTN) to separate the target feature space from the original source space while aligned with a transformed source space. Moreover, we present a non-parametric multi-class entropy minimization loss to further boost the discriminative power of FTNs on the target domain. In experiments, we first illustrate how FTN works in a controlled setting of adapting from MNIST-M to MNIST with disjoint digit classes between the two domains and then demonstrate the effectiveness of FTNs through state-of-the-art performances on a cross-ethnicity face recognition problem.

Publication Link: https://openreview.net/pdf?id=BklhAj09K7

Learning To Simulate

May 6, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 5/6/2019

Event: Seventh International Conference on Learning Representations (ICLR 2019)

Reference: pp. 1-12, 2019

Authors: Nataniel Ruiz, Boston University, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Simulation is a useful tool in situations where training data for machine learning models is costly to annotate or even hard to acquire. In this work, we propose a reinforcement learning-based method for automatically adjusting the parameters of any (non-differentiable) simulator, thereby controlling the distribution of synthesized data in order to maximize the accuracy of a model trained on that data. In contrast to prior art that hand-crafts these simulation parameters or adjusts only parts of the available parameters, our approach fully controls the simulator with the actual underlying goal of maximizing accuracy, rather than mimicking the real data distribution or randomly generating a large volume of data. We find that our approach (i) quickly converges to the optimal simulation parameters in controlled experiments and (ii) can indeed discover good sets of parameters for an image rendering simulator in actual computer vision applications.

Publication Link: https://openreview.net/pdf?id=HJgkx2Aqt7

Learning 2D to 3D Lifting for Object Detection in 3D for Autonomous Vehicles (arXiv)

April 23, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 4/23/2019

Event: arXiv

Reference: https://arxiv.org/abs/1904.08494

Authors: Siddharth Srivastava , IIT Delhi; Frederic Jurie , University of Caen Normandie; Gaurav Sharma, NEC Laboratories America, Inc.

Abstract: We address the problem of 3D object detection from 2D monocular images in autonomous driving scenarios. We propose to lift the 2D images to 3D representations using learned neural networks and leverage existing networks working directly on 3D data to perform 3D object detection and localization. We show that, with carefully designed training mechanism and automatically selected minimally noisy data, such a method is not only feasible, but gives higher results than many methods working on actual 3D inputs acquired from physical sensors. On the challenging KITTI benchmark, we show that our 2D to 3D lifted method outperforms many recent competitive 3D networks while significantly outperforming previous state-of-the-art for 3D detection from monocular images. We also show that a late fusion of the output of the network trained on generated 3D images, with that trained on real 3D images, improves performance. We find the results very interesting and argue that such a method could serve as a highly reliable backup in case of malfunction of expensive 3D sensors, if not potentially making them redundant, at least in the case of low human injury risk autonomous navigation scenarios like warehouse automation.

Publication Link: https://arxiv.org/pdf/1904.08494.pdf

Active Adversarial Domain Adaptation (arXiv)

April 16, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 4/16/2019

Event: arXiv

Reference: https://arxiv.org/abs/1904.07848v1

Authors: Jong-Chyi Su, University of Massachusetts, Amherst, NEC Laboratories America, Inc.; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Buyu Liu, NEC Laboratories America, Inc.; Subhransu Maji, University of Massachusetts, Amherst; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: We propose an active learning approach for transferring representations across domains. Our approach, active adversarial domain adaptation (AADA), explores a duality between two related problems: adversarial domain alignment and importance sampling for adapting models across domains. The former uses a domain discriminative model to align domains, while the latter utilizes it to weigh samples to account for distribution shifts. Specifically, our importance weight promotes samples with large uncertainty in classification and diversity from labeled examples, thus serves as a sample selection scheme for active learning. We show that these two views can be unified in one framework for domain adaptation and transfer learning when the source domain has many labeled examples while the target domain does not. AADA provides significant improvements over fine-tuning based approaches and other sampling methods when the two domains are closely related. Results on challenging domain adaptation tasks, e.g., object detection, demonstrate that the advantage over baseline approaches is retained even after hundreds of examples being actively annotated.

Publication Link: https://arxiv.org/pdf/1904.07848v1.pdf

Attentive Conditional Channel-Recurrent Autoencoding for Attribute-Conditioned Face Synthesis

January 8, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 1/8/2019

Event: Winter Conference on Applications of Computer Vision (WACV) 2019, Waikoloa Village, Hawaii USA

Reference: pp 1533-1542, 2019

Authors: Wenling Shang, UvA-Bosch-Deltalab, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.

Abstract: Attribute-conditioned face synthesis has many potential use cases, such as to aid the identification of a suspect or a missing person. Building on top of a conditional version of VAE-GAN, we augment the pathways connecting the latent space with channel-recurrent architecture, in order to provide not only improved generation qualities but also interpretable high-level features. In particular, to better achieve the latter, we further propose an attention mechanism over each attribute to indicate the specific latent subset responsible for its modulation. Thanks to the latent semantics formed via the channel-recurrency, we envision a tool that takes the desired attributes as inputs and then performs a 2-stage general-to-specific generation of diverse and realistic faces. Lastly, we incorporate the progressive-growth training scheme to the inference, generation and discriminator networks of our models to facilitate higher resolution outputs. Evaluations are performed through both qualitative visual examination and quantitative metrics, namely inception scores, human preferences, and attribute classification accuracy.

Publication Link: https://ieeexplore.ieee.org/document/8659125

Memory Warps for Long-Term Online Video Representations and Anticipation

January 8, 2019/0 Comments/in Publications/by NEC Labs America

Publication Date: 1/8/2019

Event: Winter Conference on Applications of Computer Vision (WACV) 2019, Waikoloa Village, Hawaii USA

Reference: pp 1156-1165, 2019

Authors: Tuan-Hung Vu, Ecole normale supérieure, NEC Laboratories America, Inc.; Wongun Choi, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: We propose a novel memory-based online video representation that is efficient, accurate and predictive. This is in contrast to prior works that often rely on computationally heavy 3D convolutions, ignore motion when aligning features over time, or operate in an off-line mode to utilize future frames. In particular, our memory (i) holds the feature representation, (ii) is spatially warped over time to compensate for observer and scene motions, (iii) can carry long-term information, and (iv) enables predicting feature representations in future frames. By exploring a variant that operates at multiple temporal scales, we efficiently learn across even longer time horizons. We apply our online framework to object detection in videos, obtaining a large 2.3 times speed-up and losing only 0.9% mAP on ImageNet-VID dataset, compared to prior works that even use future frames. Finally, we demonstrate the predictive property of our representation in two novel detection setups, where features are propagated over time to (i) significantly enhance a real-time detector by more than 10% mAP in a multi-threaded online setup and to (ii) anticipate objects in future frames.

Publication Link: https://ieeexplore.ieee.org/document/8658558

Scalable Deep k-Subspace Clustering

December 2, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 12/2/2018

Event: ACCV 2018, Perth, Australia

Reference: pp 466-481, 2019

Authors: Tong Zhang, Australian National University; Pan Ji, NEC Labs America and University of Adelaide; Mehrtash Harandi, Australian National University, Data61-CSIRO and Monash University; Richard Hartley, Australian National University; Ian Reid, University of Adelaide

Abstract: Subspace clustering algorithms are notorious for their scalability issues because building and processing large affinity matrices are demanding. In this paper, we introduce a method that simultaneously learns an embedding space along subspaces within it to minimize a notion of reconstruction error, thus addressing the problem of subspace clustering in an end-to-end learning paradigm. To achieve our goal, we propose a scheme to update subspaces within a deep neural network. This in turn frees us from the need of having an affinity matrix to perform clustering. Unlike previous attempts, our method can easily scale up to large datasets, making it unique in the context of unsupervised learning with deep architectures. Our experiments show that our method significantly improves the clustering accuracy while enjoying cheaper memory footprints.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-030-20873-8_30

Unseen Object Segmentation in Videos via Transferable Representations

December 2, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 12/2/2018

Event: ACCV 2018

Reference: pp 615-631, 2019

Authors: Yi-Wen Chen , Academia Sinica; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Chu-Ya Yang , Academia Sinica; Yen-Yu Lin , Academia Sinica; Ming-Hsuan Yang , UC Merced

Abstract: In order to learn object segmentation models in videos, conventional methods require a large amount of pixel-wise ground truth annotations. However, collecting such supervised data is time-consuming and labor-intensive. In this paper, we exploit existing annotations in source images and transfer such visual information to segment videos with unseen object categories. Without using any annotations in the target video, we propose a method to jointly mine useful segments and learn feature representations that better adapt to the target frames. The entire process is decomposed into two tasks: (1) solving a submodular function for selecting object-like segments, and (2) learning a CNN model with a transferable module for adapting seen categories in the source domain to the unseen target video. We present an iterative update scheme between two tasks to self-learn the final solution for object segmentation. Experimental results on numerous benchmark datasets show that the proposed method performs favorably against the state-of-the-art algorithms.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-030-20870-7_38

Learning Gibbs-Regularized Pushforward Density Estimators with a Symmetric KL Objective

October 11, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/11/2018

Event: BayLearn Symposium 2018, Menlo Park, CA USA

Reference: PP 1-3, 2018

Authors: Nicholas Rhinehart, The Robotics Institute, Carnegie Mellon University, NEC Laboratories America, Inc.; Anqi Liu, Department of Computer Science, University of Illinois at Chicago, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Paul Vernaza, NEC Laboratories America, Inc.

Abstract: We claim that there is currently no satisfactory way to regularize a generative adversarial network (GAN): neither the generator nor discriminator is particularly amenable to the imposition of inductive biases derived from domain knowledge. A generator is effectively a causal model of generationone that usually bears no resemblance to the true generation process, which is most often unobserved or exceedingly difficult to model. Consider image generation: although it is plausiblee.g., from biological argumentsthat convolutional neural networks constitute a good class of image classifiers, claiming CNNs are inherently well-suited to image generation is harder to justify. Likewise, it is clear that regularizing the discriminator is necessary to prevent trivial solutions; although recent methods have seen some success in applying generic smoothness regularizers to the discriminator [1, 5, 12], it is not obvious how to impose domain-specific structure on the discriminator in an optimal way

Publication Link:

Unsupervised Cross Domain Distance Metric Adaptation with Feature Transfer Network

October 11, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/11/2018

Event: BayLearn Symposium 2018, Menlo Park, CA USA

Reference: pp 1-13, 2018

Authors: Kihyuk Sohn, NEC Laboratories America, Inc.; Wenling Shang, University of Amsterdam; Xiang Yu, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Unsupervised domain adaptation is an attractive avenue to enhance the performance of deep neural networks in a target domain, using labels only from a source domain. However, two predominant methods along this line, namely, domain divergence reduction learning and semi-supervised learning, are not readily applicable when the source and target domains do not share a common label space. This paper addresses the above scenario by learning a representation space that retains discriminative power on both the (labeled) source and (unlabeled) target domains while keeping the representations for the two domains well-separated. Inspired by a theoretical error bound on the target domain, we first reformulate the disjoint classification, where the source and target domains correspond to non-overlapping class labels, to a verification task. To handle both within-domain and cross-domain verification tasks, we propose a Feature Transfer Network (FTN) that separates the target features from the source features while simultaneously aligning the target features with a transformed source feature space. Moreover, we present a non-parametric variation of multi-class entropy minimization loss to further boost the discriminative power of FTNs on the target domain. In experiments, we demonstrate the effectiveness of FTNs through state-of-the-art performances on a cross-ethnicity face recognition problem.

Publication Link:

Deep Supervision with Intermediate Concepts (BayLearn)

October 11, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/11/2018

Event: BayLearn Symposium 2018, Menlo Park, CA USA

Reference: pp 1-3, 2018

Authors: Chi Li, Johns Hopkins University, NEC Laboratories America, Inc.; M. Zeeshan Zia, NEC Laboratories America, Inc.; Quoc-Huy Tran, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Gregory Hager, Johns Hopkins University; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: We introduce a novel technique for training convolutional neural networks (CNNs), namely deep supervision with intermediate concepts, leading to improved generalization. Our approach draws inspiration from Deeply Supervised Nets (DSN) [5], which supervises each layer by the main task to accelerate training convergence. Our method differs from DSN in that we apply deep supervision with intermediate concepts, intrinsic to the ultimate task, to regularize the network for better generalization. We apply this improved generalization ability to transfer knowledge from synthetic to real images.

Publication Link:

Learning to Simulate (arXiv)

October 5, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 10/5/2018

Event: arXiv

Reference: https://arxiv.org/abs/1810.02513v1

Authors: Nataniel Ruiz, Boston University, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Simulation is a useful tool in situations where training data for machine learning models is costly to annotate or even hard to acquire. In this work, we propose a reinforcement learning-based method for automatically adjusting the parameters of any (non-differentiable) simulator, thereby controlling the distribution of synthesized data in order to maximize the accuracy of a model trained on that data. In contrast to prior art that handcrafts these simulation parameters or adjusts only parts of the available parameters, our approach fully controls the simulator with the actual underlying goal of maximizing accuracy, rather than mimicking the real data distribution or randomly generating a large volume of data. We find that our approach (i) quickly converges to the optimal simulation parameters in controlled experiments and (ii) can indeed discover good sets of parameters for an image rendering simulator in actual computer vision applications.

Publication Link: https://arxiv.org/pdf/1810.02513v1.pdf

Hierarchical Metric Learning and Matching for 2D and 3D Geometric Correspondences

September 8, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 9/8/2018

Event: European Conference on Computer Vision – ECCV 2018, Munich, Germany

Reference: pp 832-850, 2018

Authors: Mohammed E. Fathy, Google, NEC Laboratories America, Inc.; Quoc-Huy Tran, NEC Laboratories America, Inc.; M. Zeeshan Zia, Microsoft, NEC Laboratories America, Inc.; Paul Vernaza, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., University of California

Abstract: Interest point descriptors have fueled progress on almost every problem in computer vision. Recent advances in deep neural networks have enabled task-specific learned descriptors that outperform hand-crafted descriptors on many problems. We demonstrate that commonly used metric learning approaches do not optimally leverage the feature hierarchies learned in a Convolutional Neural Network (CNN), especially when applied to the task of geometric feature matching. While a metric loss applied to the deepest layer of a CNN, is often expected to yield ideal features irrespective of the task, in fact the growing receptive field as well as striding effects cause shallower features to be better at high precision matching tasks. We leverage this insight together with explicit supervision at multiple levels of the feature hierarchy for better regularization, to learn more effective descriptors in the context of geometric matching tasks. Further, we propose to use activation maps at different layers of a CNN, as an effective and principled replacement for the multi-resolution image pyramids often used for matching tasks. We propose concrete CNN architectures employing these ideas and evaluate them on multiple datasets for 2D and 3D geometric matching as well as optical flow, demonstrating state-of-the-art results and generalization across datasets.

Publication Link: https://openaccess.thecvf.com/content_ECCV_2018/html/Mohammed_Fathy_Hierarchical_Metric_Learning_ECCV_2018_paper.html

R2P2: A Reparameterized Pushforward Policy for Diverse, Precise Generative Path Forecasting

September 8, 2018/in Publications/by NEC Labs America

Publication Date: 9/8/2018

Event: European Conference on Computer Vision – ECCV 2018, Munich, Germany

Reference: pp 794-811, 2018

Authors: Nicholas Rhinehart, Carnegie-Mellon University and NEC Laboratories America, Inc.; Kris M. Kitani, Carnegie-Mellon University; Paul Vernaza, NEC Laboratories America, Inc.

Abstract: We propose a method to forecast a vehicle’s ego-motion as a distribution over spatiotemporal paths, conditioned on features (e.g., from LIDAR and images) embedded in an overhead map. The method learns a policy inducing a distribution over simulated trajectories that is both diverse (produces most paths likely under the data) and precise (mostly produces paths likely under the data). This balance is achieved through minimization of a symmetrized cross-entropy between the distribution and demonstration data. By viewing the simulated-outcome distribution as the pushforward of a simple distribution under a simulation operator, we obtain expressions for the cross-entropy metrics that can be efficiently evaluated and differentiated, enabling stochastic-gradient optimization. We propose concrete policy architectures for this model, discuss our evaluation metrics relative to previously-used metrics, and demonstrate the superiority of our method relative to state-of-the-art methods in both the KITTI dataset and a similar but novel and larger real-world dataset explicitly designed for the vehicle forecasting domain.

Publication Link: https://openaccess.thecvf.com/content_ECCV_2018/html/Nicholas_Rhinehart_R2P2_A_ReparameteRized_ECCV_2018_paper.html

Supplemental Publication Link: https://nec-labs.com/R2P2-_A_Reparameterized_Pushforward_Policy_for_Diverse_Precise_Generative_Path_5AWGIU4.pdf

Learning to Look around Objects for Top-View Representations of Outdoor Scenes (arXiv)

September 8, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 9/8/2018

Event: European Conference on Computer Vision – ECCV 2018, Munich, Germany

Reference: pp 815-831, 2018

Authors: Samuel Schulter, NEC Laboratories America, Inc.; Menghua Zhai, University of Kentucky, NEC Laboratories America, Inc.; Nathan Jacobs, University of Kentucky; Manmohan Chandraker, NEC Laboratories America, Inc., University of California

Abstract: Given a single RGB image of a complex outdoor road scene in the perspective view, we address the novel problem of estimating an occlusion-reasoned semantic scene layout in the top-view. This challenging problem not only requires an accurate understanding of both the 3D geometry and the semantics of the visible scene, but also of occluded areas. We propose a convolutional neural network that learns to predict occluded portions of the scene layout by looking around foreground objects like cars or pedestrians. But instead of hallucinating RGB values, we show that directly predicting the semantics and depths in the occluded areas enables a better transformation into the top-view. We further show that this initial top-view representation can be significantly enhanced by learning priors and rules about typical road layouts from simulated or, if available, map data. Crucially, training our model does not require costly or subjective human annotations for occluded areas or the top-view, but rather uses readily available annotations for standard semantic segmentation in the perspective view. We extensively evaluate and analyze our approach on the KITTI and Cityscapes data sets.

Publication Link: https://openaccess.thecvf.com/content_ECCV_2018/html/Samuel_Schulter_Learning_to_Look_ECCV_2018_paper.html

Zero-Shot Object Detection

September 8, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 9/8/2018

Event: European Conference on Computer Vision – ECCV 2018, Munich, Germany

Reference: pp 397-141, 2018

Authors: Ankan Bansal, University of Maryland; Karan Sikka , SRI International; Gaurav Sharma , NEC Laboratories America, Inc.; Rama Chellappa , University of Maryland; Ajay Divakaran , SRI International

Abstract: We introduce and tackle the problem of zero-shot object detection (ZSD), which aims to detect object classes which are not observed during training. We work with a challenging set of object classes, not restricting ourselves to similar and/or fine-grained categories as in prior works on zero-shot classification. We present a principled approach by first adapting visual-semantic embeddings for ZSD. We then discuss the problems associated with selecting a background class and motivate two background-aware approaches for learning robust detectors. One of these models uses a fixed background class and the other is based on iterative latent assignments. We also outline the challenge associated with using a limited number of training classes and propose a solution based on dense sampling of the semantic label space using auxiliary data with a large number of categories. We propose novel splits of two standard detection datasets MSCOCO and VisualGenome, and present extensive empirical results in both the traditional and generalized zero-shot settings to highlight the benefits of the proposed methods. We provide useful insights into the algorithm and conclude by posing some open questions to encourage further research.

Publication Link: https://openaccess.thecvf.com/content_ECCV_2018/html/Ankan_Bansal_Zero-Shot_Object_Detection_ECCV_2018_paper.html

Fast and Accurate Online Video Object Segmentation via Tracking Parts

June 18, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 6/18/2018

Event: Conference on Computer Vision and Pattern Recognition (CVPR) 2018, Salt Lake City, UT USA

Reference: pp 7415-7424, 2018

Authors: Jingchun Cheng, Tsinghua University, China, University of California, Merced; Yi-Hsuan Tsai, University of California, Merced, NEC Laboratories America, Inc.; Wei-Chih Hung, University of California, Merced; Shengjin Wang, Tsinghua University, China; Ming-Hsuan Yang, University of California, Merced, NEC Laboratories America, Inc.

Abstract: Online video object segmentation is a challenging task as it entails to process the image sequence timely and accurately. To segment a target object through the video, numerous CNN-based methods have been developed by heavily finetuning on the object mask in the first frame, which is time-consuming for online applications. In this paper, we propose a fast and accurate video object segmentation algorithm that can immediately start the segmentation process once receiving the images. We first utilize a part-based tracking method to deal with challenging factors such as large deformation, occlusion, and cluttered background. Based on the tracked bounding boxes of parts, we construct a region-of-interest segmentation network to generate part masks. Finally, a similarity-based scoring function is adopted to refine these object parts by comparing them to the visual information in the first frame. Our method performs favorably against state-of-the-art algorithms in accuracy on the DAVIS benchmark dataset, while achieving much faster runtime performance.

Publication Link: https://openaccess.thecvf.com/content_cvpr_2018/html/Cheng_Fast_and_Accurate_CVPR_2018_paper.html

Learning to Adapt Structured Output Space for Semantic Segmentation

June 18, 2018/in Publications/by NEC Labs America

Publication Date: 6/18/2018

Event: Conference on Computer Vision and Pattern Recognition (CVPR) 2018, Salt Lake City, UT USA

Reference: pp 7472-7481

Authors: Yi-Hsuan Tsai, University of California, Merced, NEC Laboratories America, Inc.; Wei-Chih Hung, University of California, Merced; Samuel Schulter, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Ming-Hsuan Yang, University of California, Merced, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Convolutional neural network-based approaches for semantic segmentation rely on supervision with pixel-level ground truth, but may not generalize well to unseen image domains. As the labeling process is tedious and labor intensive, developing algorithms that can adapt source ground truth labels to the target domain is of great interest. In this paper, we propose an adversarial learning method for domain adaptation in the context of semantic segmentation. Considering semantic segmentations as structured outputs that contain spatial similarities between the source and target domains, we adopt adversarial learning in the output space. To further enhance the adapted model, we construct a multi-level adversarial network to effectively perform output space domain adaptation at different feature levels. To further improve our method, we utilize multi-level output adaptation based on feature maps at different levels. Extensive experiments and ablation study are conducted under various domain adaptation settings, including synthetic-to-real and cross-city scenarios. We show that the proposed method performs favorably against the state-of-the-art methods in terms of accuracy and visual quality.

Publication Link: https://openaccess.thecvf.com/content_cvpr_2018/html/Tsai_Learning_to_Adapt_CVPR_2018_paper.html

Supplemental Publication Link: https://openaccess.thecvf.com/content_cvpr_2018/Supplemental/1435-supp.pdf

Zero-Shot Object Detection (arXiv)

April 12, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 4/12/2018

Event: arXiv

Reference: https://arxiv.org/abs/1804.04340v1

Authors: Ankan Bansal, University of Maryland, College Park; Karan Sikka, SRI International; Gaurav Sharma, NEC Laboratories America, Inc.; Rama Chellappa, University of Maryland, College Park; Ajay Divakaran, SRI International

Abstract: We introduce and tackle the problem of zero-shot object detection (ZSD), which aims to detect object classes which are not observed during training. We work with a challenging set of object classes, not restricting ourselves to similar and/or fine-grained categories cf. prior works on zero-shot classification. We follow a principled approach by first adapting visual-semantic embeddings for ZSD. We then discuss the problems associated with selecting a background class and motivate two background-aware approaches for learning robust detectors. One of these models uses a fixed background class and the other is based on iterative latent assignments. We also outline the challenge associated with using a limited number of training classes and propose a solution based on dense sampling of the semantic label space using auxiliary data with a large number of categories. We propose novel splits of two standard detection datasets – MSCOCO and VisualGenome and discuss extensive empirical results to highlight the benefits of the proposed methods. We provide useful insights into the algorithm and conclude by posing some open questions to encourage further research.

Publication Link: https://arxiv.org/pdf/1804.04340v1.pdf

Learning to Look around Objects for Top-View Representations of Outdoor Scenes

March 28, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 3/28/2018

Event: arXiv

Reference: https://arxiv.org/abs/1803.10870v1

Authors: Samuel Schulter, NEC Laboratories America, Inc.; Menghua Zhai, University of Kentucky, NEC Laboratories America, Inc.; Nathan Jacobs, Computer Science, University of Kentucky; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Given a single RGB image of a complex outdoor road scene in the perspective view, we address the novel problem of estimating an occlusion-reasoned semantic scene layout in the top-view. This challenging problem not only requires an accurate understanding of both the 3D geometry and the semantics of the visible scene, but also of occluded areas. We propose a convolutional neural network that learns to predict occluded portions of the scene layout by looking around foreground objects like cars or pedestrians. But instead of hallucinating RGB values, we show that directly predicting the semantics and depths in the occluded areas enables a better transformation into the top-view. We further show that this initial top-view representation can be significantly enhanced by learning priors and rules about typical road layouts from simulated or, if available, map data. Crucially, training our model does not require costly or subjective human annotations for occluded areas or the top-view, but rather uses readily available annotations for standard semantic segmentation. We extensively evaluate and analyze our approach on the KITTI and Cityscapes data sets.

Publication Link: https://arxiv.org/pdf/1803.10870v1.pdf

Memory Warps for Learning Long-Term Online Video Representations

March 28, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 3/28/2018

Event: arXiv

Reference: https://arxiv.org/abs/1803.10861v1

Authors: Tuan-Hung Vu, Ecole normale supérieure, NEC Laboratories America, Inc.; Wongun Choi, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: This paper proposes a novel memory-based online video representation that is efficient, accurate and predictive. This is in contrast to prior works that often rely on computationally heavy 3D convolutions, ignore actual motion when aligning features over time, or operate in an off-line mode to utilize future frames. In particular, our memory (i) holds the feature representation, (ii) is spatially warped over time to compensate for observer and scene motions, (iii) can carry long-term information, and (iv) enables predicting feature representations in future frames. By exploring a variant that operates at multiple temporal scales, we efficiently learn across even longer time horizons. We apply our online framework to object detection in videos, obtaining a large 2.3 times speed-up and losing only 0.9% mAP on ImageNet-VID dataset, compared to prior works that even use future frames. Finally, we demonstrate the predictive property of our representation in two novel detection setups, where features are propagated over time to (i) significantly enhance a real-time detector by more than 10% mAP in a multi-threaded online setup and to (ii) anticipate objects in future frames.

Publication Link: https://arxiv.org/pdf/1803.10861v1.pdf

Feature Transfer Learning for Deep Face Recognition with Long-Tail Data

March 23, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 3/23/2018

Event: arXiv

Reference: https://arxiv.org/abs/1803.09014v1

Authors: Xi Yin, Michigan State University, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Xiaoming Liu, Michigan State University; Manmohan Chandraker, NEC Laboratories America, Inc., University of California, San Diego

Abstract: Real-world face recognition datasets exhibit long-tail characteristics, which results in biased classifiers in conventionally-trained deep neural networks, or insufficient data when long-tail classes are ignored. In this paper, we propose to handle long-tail classes in the training of a face recognition engine by augmenting their feature space under a center-based feature transfer framework. A Gaussian prior is assumed across all the head (regular) classes and the variance from regular classes are transferred to the long-tail class representation. This encourages the long-tail distribution to be closer to the regular distribution, while enriching and balancing the limited training data. Further, an alternating training regimen is proposed to simultaneously achieve less biased decision boundaries and a more discriminative feature representation. We conduct empirical studies that mimic long-tail datasets by limiting the number of samples and the proportion of long-tail classes on the MS-Celeb-1M dataset. We compare our method with baselines not designed to handle long-tail classes and also with state-of-the-art methods on face recognition benchmarks. State-of-the-art results on LFW, IJB-A and MS-Celeb-1M datasets demonstrate the effectiveness of our feature transfer approach and training strategy. Finally, our feature transfer allows smooth visual interpolation, which demonstrates disentanglement to preserve identity of a class while augmenting its feature space with non-identity variations.

Publication Link: https://arxiv.org/pdf/1803.09014v1.pdf

Channel-Recurrent Autoencoding for Image Modeling

March 14, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 3/14/2018

Event: WACV 2018, Lake Tahoe, Nevada USA

Reference: pp 1195-1204, 2018

Authors: Wenling Shang , University of Amsterdam; Kihyuk Sohn , NEC Laboratories America, Inc.; Yuandong Tian , Facebook AI Research

Abstract: Despite recent successes in synthesizing faces and bedrooms, existing generative models struggle to capture more complex image types (Figure 1), potentially due to the oversimplification of their latent space constructions. To tackle this issue, building on Variational Autoencoders (VAEs), we integrate recurrent connections across channels to both inference and generation steps, allowing the high-level features to be captured in global-to-local, coarse-to-fine manners. Combined with adversarial loss, our channel-recurrent VAE-GAN (crVAE-GAN) outperforms VAE-GAN in generating a diverse spectrum of high resolution images while maintaining the same level of computational efficacy. Our model produces interpretable and expressive latent representations to benefit downstream tasks such as image completion. Moreover, we propose two novel regularizations, namely the KL objective weighting scheme over time steps and mutual information maximization between transformed latent variables and the outputs, to enhance the training.

Publication Link: https://ieeexplore.ieee.org/document/8354240

SVBRDF-Invariant Shape and Reflectance Estimation from a Light-Field Camera

February 22, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 2/22/2018

Event: IEEE Transactions on Pattern Analysis and Machine Intelligence

Reference: Vol. 40, No. 3, pp 740-754

Authors: Ting-Chun Wang, University of California, Berkeley; Manmohan Chandraker, NEC Laboratories America, Inc.; Alexei A. Efros, University of California, Berkeley; Ravi Ramamoorthi, University of California, San Diego

Abstract: Light-field cameras have recently emerged as a powerful tool for one-shot passive 3D shape capture. However, obtaining the shape of glossy objects like metals or plastics remains challenging, since standard Lambertian cues like photo-consistency cannot be easily applied. In this paper, we derive a spatially-varying (SV)BRDF-invariant theory for recovering 3D shape and reflectance from light-field cameras. Our key theoretical insight is a novel analysis of diffuse plus single-lobe SVBRDFs under a light-field setup. We show that, although direct shape recovery is not possible, an equation relating depths and normals can still be derived. Using this equation, we then propose using a polynomial (quadratic) shape prior to resolve the shape ambiguity. Once shape is estimated, we also recover the reflectance. We present extensive synthetic data on the entire MERL BRDF dataset, as well as a number of real examples to validate the theory, where we simultaneously recover shape and BRDFs from a single image taken with a Lytro Illum camera.

Publication Link: https://ieeexplore.ieee.org/document/7875163

Joint Pixel and Feature-level Domain Adaptation in the Wild

February 5, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 2/5/2018

Event: arXiv

Reference: https://arxiv.org/abs/1803.00068v1

Authors: Luan Tran, Michigan State University, NEC Laboratories America, Inc.; Kihyuk Sohn, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Xiaoming Liu, Michigan State University; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: Recent developments in deep domain adaptation have allowed knowledge transfer from a labeled source domain to an unlabeled target domain at the level of intermediate features or input pixels. We propose that advantages may be derived by combining them, in the form of different insights that lead to a novel design and complementary properties that result in better performance. At the feature level, inspired by insights from semi-supervised learning in a domain adversarial neural network, we propose a novel regularization in the form of domain adversarial entropy minimization. Next, we posit that insights from computer vision are more amenable to injection at the pixel level and specifically address the key challenge of adaptation across different semantic levels. In particular, we use 3D geometry and image synthetization based on a generalized appearance flow to preserve identity across higher-level pose transformations, while using an attribute-conditioned CycleGAN to translate a single source into multiple target images that differ in lower-level properties such as lighting. We validate on a novel problem of car recognition in unlabeled surveillance images using labeled images from the web, handling explicitly specified, nameable factors of variation through pixel-level and implicit, unspecified factors through feature-level adaptation. Extensive experiments achieve state-of-the-art results, demonstrating the effectiveness of complementing feature and pixel-level information via our proposed domain adaptation method.

Publication Link: https://arxiv.org/pdf/1803.00068v1.pdf

Learning random-walk label propagation for weakly-supervised semantic segmentation

February 1, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 2/1/2018

Event: arXiv

Reference: https://arxiv.org/abs/1802.00470v1

Authors: Paul Vernaza, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Large-scale training for semantic segmentation is challenging due to the expense of obtaining training data for this task relative to other vision tasks. We propose a novel training approach to address this difficulty. Given cheaply-obtained sparse image labelings, we propagate the sparse labels to produce guessed dense labelings. A standard CNN-based segmentation network is trained to mimic these labelings. The label-propagation process is defined via random-walk hitting probabilities, which leads to a differentiable parameterization with uncertainty estimates that are incorporated into our loss. We show that by learning the label-propagator jointly with the segmentation predictor, we are able to effectively learn semantic edges given no direct edge supervision. Experiments also show that training a segmentation network in this way outperforms the naive approach.

Publication Link: https://arxiv.org/pdf/1802.00470v1.pdf

Deep Supervision with Intermediate Concepts (arXiv)

January 8, 2018/0 Comments/in Publications/by NEC Labs America

Publication Date: 1/8/2018

Event: arXiv

Reference: https://arxiv.org/abs/1801.03399v1

Authors: Chi Li, Johns Hopkins University, NEC Laboratories America, Inc.; M. Zeeshan Zia, NEC Laboratories America, Inc.; Quoc-Huy Tran, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Gregory D. Hager, Johns Hopkins University; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Recent data-driven approaches to scene interpretation predominantly pose inference as an end-to-end black-box mapping, commonly performed by a Convolutional Neural Network (CNN). However, decades of work on perceptual organization in both human and machine vision suggests that there are often intermediate representations that are intrinsic to an inference task, and which provide essential structure to improve generalization. In this work, we explore an approach for injecting prior domain structure into neural network training by supervising hidden layers of a CNN with intermediate concepts that normally are not observed in practice. We formulate a probabilistic framework which formalizes these notions and predicts improved generalization via this deep supervision method. One advantage of this approach is that we are able to train only from synthetic CAD renderings of cluttered scenes, where concept values can be extracted, but apply the results to real images. Our implementation achieves the state-of-the-art performance of 2D/3D keypoint localization and image classification on real image benchmarks, including KITTI, PASCAL VOC, PASCAL3D+, IKEA, and CIFAR100. We provide additional evidence that our approach outperforms alternative forms of supervision, such as multi-task networks.

Publication Link: https://arxiv.org/pdf/1801.03399.pdf

WarpNet: Weakly Supervised Matching for Single-View Reconstruction

June 1, 2016/in Publications/by NEC Labs America

Publication Date: 6/1/2016

Event: CVPR 2016

Authors: Angjoo Kanazawa, University of Maryland, College Park; Manmohan Chandraker, David W. Jacobs, University of Maryland, College Park

Abstract: Our WarpNet matches images of objects in fine-grained datasets without using part annotations. It aligns an object in one image with a different object in another by exploiting a fine-grained dataset to create artificial data for training a Siamese network with an unsupervised discriminative learning approach. The output of the network acts as a spatial prior that allows generalization at test time to match real images across variations in appearance, viewpoint and articulation. This allows single-view reconstruction with quality comparable to using human annotation.

Publication Link: https://openaccess.thecvf.com/content_cvpr_2016/papers/Kanazawa_WarpNet_Weakly_Supervised_CVPR_2016_paper.pdf

Supplemental Publication Link: https://openaccess.thecvf.com/content_cvpr_2016/supplemental/Kanazawa_WarpNet_Weakly_Supervised_2016_CVPR_supplemental.zip

Media Analytics | Publications

Contact Us

About Us

Our Pages

Read Our Blog Posts