Publications

Improving Pseudo Labels for Open-Vocabulary Object Detection

August 2, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 8/2/2023

Event: https://arxiv.org

Reference: https://arxiv.org/abs/2308.06412

Authors: Shiyu Zhao, Rutgers University, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Long Zhao, Google Research ; Zhixing Zhang, Rutgers University, NEC Laboratories America, Inc.; Vijay Kumar BG, NEC Laboratories America, Inc.; Yumin Suh, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Dimitris Metaxas, Rutgers University

Abstract: Recent studies show promising performance in open-vocabulary object detection (OVD) using pseudo labels (PLs) from pretrained vision and language models (VLMs). However, PLs generated by VLMs are extremely noisy due to the gap between the pretraining objective of VLMs and OVD, which blocks further advances on PLs. In this paper, we aim to reduce the noise in PLs and propose a method called online Self-training And a Split-and-fusion head for OVD (SAS-Det). First, the self-training finetunes VLMs to generate high quality PLs while prevents forgetting the knowledge learned in the pretraining. Second, a split-and-fusion (SAF) head is designed to remove the noise in localization of PLs, which is usually ignored in existing methods. It also fuses complementary knowledge learned from both precise ground truth and noisy pseudo labels to boost the performance. Extensive experiments demonstrate SAS-Det is both efficient and effective. Our pseudo labeling is 3 times faster than prior methods. SAS-Det outperforms prior state-of-the-art models of the same scale by a clear margin and achieves 37.4 AP50 and 27.3 APr on novel categories of the COCO and LVIS benchmarks, respectively.

Publication Link: https://arxiv.org/pdf/2308.06412.pdf

Confidence and Dispersity Speak: Characterizing Prediction Matrix for Unsupervised Accuracy Estimation

July 23, 2023/0 Comments/in Publications/by NEC Labs America

Publication Date: 7/23/2023

Event: ICML 2023

Reference: Honolulu, Hawai’i

Authors: Weijian Deng, Australian National University; Yumin Suh, NEC Laboratories America, Inc.; Stephen Gould, Australian National University; Liang Zheng, Australian National University

Abstract: This work aims to assess how well a model performs under distribution shifts without using labels. While recent methods study prediction confidence, this work reports prediction dispersity is another informative cue. Confidence reflects whether the individual prediction is certain; dispersity indicates how the overall predictions are distributed across all categories. Our key insight is that a well-performing model should give predictions with high confidence and high dispersity. That is, we need to consider both properties so as to make more accurate estimates. To this end, we use the nuclear norm that has been shown to be effective in characterizing both properties. Extensive experiments validate the effectiveness of nuclear norm for various models (e.g., ViT and ConvNeXt), different datasets (e.g., ImageNet and CUB-200), and diverse types of distribution shifts (e.g., style shift and reproduction shift). We show that the nuclear norm is more accurate and robust in accuracy estimation than existing methods. Furthermore, we validate the feasibility of other measurements (e.g., mutual information maximization) for characterizing dispersity and confidence. Lastly, we investigate the limitation of the nuclear norm, study its improved variant under severe class imbalance, and discuss potential directions.

Publication Link: https://icml.cc/virtual/2023/papers.html?filter=titles&search=Confidence+and+Dispersity+Speak:+Characterizing+Prediction+Matrix+for+Unsupervised+Accuracy+Estimation

Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!

June 18, 2023/in Publications/by NEC Labs America

Publication Date: 6/18/2023

Event: CVPR 2023

Reference: pp. 15005-15015

Authors: Zaid Khan, Northeastern University, NEC Laboratories America, Inc.; Vijay Kumar B G,, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Yun Fu, Northeastern University; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Finetuning a large vision language model (VLM) on a target dataset after large-scale pretraining is a dominant paradigm in visual question answering (VQA). Datasets for specialized tasks such as knowledge-based VQA or VQA in non natural-image domains are orders of magnitude smaller than those for general-purpose VQA. While collecting additional labels for specialized tasks or domains can be challenging, unlabeled images are often available. We introduce SelTDA (Self-Taught Data Augmentation), a strategy for finetuning large VLMs on small-scale VQA datasets. SelTDA uses the VLM and target dataset to build a teacher model that can generate question-answer pseudolabels directly conditioned on an image alone, allowing us to pseudolabel unlabeled images. SelTDA then finetunes the initial VLM on the original dataset augmented with freshly pseudolabeled images. We describe a series of experiments showing that our self-taught data augmentation increases robustness to adversarially searched questions, counterfactual examples, and rephrasings, it improves domain generalization, and results in greater retention of numerical reasoning skills. The proposed strategy requires no additional annotations or architectural modifications, and is compatible with any modern encoder-decoder multimodal transformer. Code available at https://github.com/codezakh/SelTDA

Publication Link: https://openaccess.thecvf.com/content/CVPR2023/html/Khan_Q_How_To_Specialize_Large_Vision-Language_Models_to_Data-Scarce_VQA_CVPR_2023_paper.html

NeurOCS: Neural NOCS Supervision for Monocular 3D Object Localization

June 18, 2023/in Publications/by NEC Labs America

Publication Date: 6/18/2023

Event: CVPR 2023

Reference: pp. 21404-21414, 2023

Authors: Zhixiang Min, Stevens Institute of Technology, NEC Laboratories America, Inc.; Bingbing Zhuang, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Buyu Liu, NEC Laboratories America, Inc.; Enrique Dunn, Stevens Institute of Technology; Manmohan Chandraker, NEC Laboratories America, Inc.

Abstract: Monocular 3D object localization in driving scenes is a crucial task, but challenging due to its ill-posed nature. Estimating 3D coordinates for each pixel on the object surface holds great potential as it provides dense 2D-3D geometric constraints for the underlying PnP problem. However, high-quality ground truth supervision is not available in driving scenes due to sparsity and various artifacts of Lidar data, as well as the practical infeasibility of collecting per-instance CAD models. In this work, we present NeurOCS, a framework that uses instance masks and 3D boxes as input to learn 3D object shapes by means of differentiable rendering, which further serves as supervision for learning dense object coordinates. Our approach rests on insights in learning a category-level shape prior directly from real driving scenes, while properly handling single-view ambiguities. Furthermore, we study and make critical design choices to learn object coordinates more effectively from an object-centric view. Altogether, our framework leads to new state-of-the-art in monocular 3D localization that ranks 1st on the KITTI-Object benchmark among published monocular methods.

Publication Link: https://openaccess.thecvf.com/content/CVPR2023/html/Min_NeurOCS_Neural_NOCS_Supervision_for_Monocular_3D_Object_Localization_CVPR_2023_paper.html

Q: How to Specialize Large Vision Language Models to Data Scarce VQA Tasks? A: Self Train on Unlabeled Images!

June 6, 2023/in Publications/by NEC Labs America

Publication Date: 6/6/2023

Event: arXiv

Reference: https://arxiv.org/pdf/2306.03932.pdf

Authors: Zaid Khan, NEC Laboratories America, Inc., Northeastern University, Vijay Kumar B G, , NEC Laboratories America, Inc., Samuel Schulter, NEC Laboratories America, Inc., Xiang Yu, NEC Laboratories America, Inc., Manmohan Chandrakar, NEC Laboratories America, Inc., Yun Fu, Northeastern University

Abstract: Finetuning a large vision language model (VLM) on a target dataset after large scale pretraining is a dominant paradigm in visual question answering (VQA). Datasets for specialized tasks such as knowledge based VQA or VQA in non natural image domains are orders of magnitude smaller than those for general purpose VQA. While collecting additional labels for specialized tasks or domains can be challenging, unlabeled images are often available. We introduce SelTDA (Self Taught Data Augmentation), a strategy for finetuning large VLMs on small scale VQA datasets. SelTDA uses the VLM and target dataset to build a teacher model that can generate question answer pseudolabels directly conditioned on an image alone, allowing us to pseudolabel unlabeled images. SelTDA then finetunes the initial VLM on the original dataset augmented with freshly pseudolabeled images. We describe a series of experiments showing that our self taught data augmentation increases robustness to adversarially searched questions, counterfactual examples and rephrasings, improves domain generalization, and results in greater retention of numerical reasoning skills. The proposed strategy requires no additional annotations or architectural modifications, and is compatible with any modern encoder decoder multimodal transformer. Code available at https://github.com/codezakh/SelTDA.

Publication Link: https://arxiv.org/pdf/2306.03932.pdf

OmniLabel: A Challenging Benchmark for Language Based Object Detection

April 4, 2023/in Publications/by NEC Labs America

Publication Date: 4/4/2023

Event: arXiv

Reference: https://arxiv.org/pdf/2304.11463.pdf

Authors: Samuel Schulter, NEC Laboratories America, Inc., Vijay Kumar B G, NEC Laboratories America, Inc., Yumin Suh, NEC Laboratories America, Inc., Konstantinos M. Dafnis, Rutgers University, Zhixing Zhang, Rutgers University, NEC Laboratories America, Inc., Shiyu Zhao, Rutgers University, NEC Laboratories America, Inc., Dimitris Metaxas, Rutgers University

Abstract: Language based object detection is a promising direction towards building a natural interface to describe objects in images that goes far beyond plain category names. While recent methods show great progress in that direction, proper evaluation is lacking. With OmniLabel, we propose a novel task definition, dataset, and evaluation metric. The task subsumes standard and open vocabulary detection as well as referring expressions. With more than 28K unique object descriptions on over 25K images, OmniLabel provides a challenging benchmark with diverse and complex object descriptions in a naturally open vocabulary setting. Moreover, a key differentiation to existing benchmarks is that our object descriptions can refer to one, multiple or even no object, hence, providing negative examples in free form text. The proposed evaluation handles the large label space and judges performance via a modified average precision metric, which we validate by evaluating strong language based baselines. OmniLabel indeed provides a challenging test bed for future research on language based detection. Visit the project website at https://www.omnilabel.org

Publication Link: https://arxiv.org/pdf/2304.11463.pdf

Split to Learn: Gradient Split for Multi-Task Human Image Analysis

January 3, 2023/in Publications/by NEC Labs America

Publication Date: 1/3/2023

Event: WACV23

Reference: pp. 4340-4346, 2023

Authors: Weijian Deng, Australian National University, NEC Laboratories America, Inc.; Yumin Suh, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Masoud Faraki, NEC Laboratories America, Inc.; Liang Zheng, Australian National University; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego

Abstract: This paper presents an approach to train a unified deep network that simultaneously solves multiple human-related tasks. A multi-task framework is favorable for sharing information across tasks under restricted computational resources. However, tasks not only share information but may also compete for resources and conflict with each other, making the optimization of shared parameters difficult and leading to suboptimal performance. We propose a simple but effective training scheme called GradSplit that alleviates this issue by utilizing asymmetric inter-task relations. Specifically, at each convolution module, it splits features into T groups for T tasks and trains each group only using the gradient back-propagated from the task losses with which it does not have conflicts. During training, we apply GradSplit to a series of convolution modules. As a result, each module is trained to generate a set of task-specific features using the shared features from the previous module. This enables a network to use complementary information across tasks while circumventing gradient conflicts. Experimental results show that GradSplit achieves a better accuracy-efficiency trade-off than existing methods. It minimizes accuracy drop caused by task conflicts while significantly saving compute resources in terms of both FLOPs and memory at inference. We further show that GradSplit achieves higher cross-dataset accuracy compared to single-task and other multi-task networks.

Publication Link: https://ieeexplore.ieee.org/document/10030388

Single-Stream Multi-level Alignment for Vision-Language Pretraining

October 24, 2022/in Publications/by NEC Labs America

Publication Date: 10/24/2022

Event: ECCV 2022

Reference: LNCS 13696, pp. 735–751, 2022

Authors: Zaid Khan, Northeastern University, NEC Laboratories America, Inc.; Vijay Kumar B G, NEC Laboratories America, Inc.; Xiang Yu, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Raymond Fu, Northeastern University

Abstract: Self-supervised vision-language pretraining from pure images and text with a contrastive loss is effective, but ignores fine-grained alignment due to a dual-stream architecture that aligns image and text representations only on a global level. Earlier, supervised, non-contrastive methods were capable of finer-grained alignment, but required dense annotations that were not scalable. We propose a single stream architecture that aligns images and language at multiple levels: global, fine-grained patch-token, and conceptual/semantic, using two novel tasks: symmetric cross-modality reconstruction (XMM) and a pseudo-labeled key word prediction (PSL). In XMM, we mask input tokens from one modality and use cross-modal information to reconstruct the masked token, thus improving fine-grained alignment between the two modalities. In PSL, we use attention to select keywords in a caption, use a momentum encoder to recommend other important keywords that are missing from the caption but represented in the image, and then train the visual encoder to predict the presence of those keywords, helping it learn semantic concepts that are essential for grounding a textual token to an image region. We demonstrate competitive performance and improved data efficiency on image-text retrieval, grounding, visual question answering/reasoning against larger models and models trained on more data. Code and models available at zaidkhan.me/SIMLA.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-031-20059-5_42

Exploiting Unlabeled Data with Vision and Language Models for Object Detection

October 24, 2022/in Publications/by NEC Labs America

Publication Date: 10/24/2022

Event: ECCV 2022

Reference: LNCS 13669, pp. 159–175, 2022

Authors: Shiyu Zhao, Rutgers University, NEC Laboratories America, Inc.; Zhixing Zhang, Rutgers University, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Long Zhao, Google Research; Vijay Kumar BG, NEC Laboratories America, Inc.; Anastasis Stathopoulos, Rutgers University; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Dimitris Metaxas, Rutgers University

Abstract: Building robust and generic object detection frameworks requires scaling to larger label spaces and bigger training datasets. However, it is prohibitively costly to acquire annotations for thousands of categories at a large scale. We propose a novel method that leverages the rich semantics available in recent vision and language models to localize and classify objects in unlabeled images, effectively generating pseudo labels for object detection. Starting with a generic and class-agnostic region proposal mechanism, we use vision and language models to categorize each region of an image into any object category that is required for downstream tasks. We demonstrate the value of the generated pseudo labels in two specific tasks, open-vocabulary detection, where a model needs to generalize to unseen object categories, and semi-supervised object detection, where additional unlabeled images can be used to improve the model. Our empirical evaluation shows the effectiveness of the pseudo labels in both tasks, where we outperform competitive baselines and achieve a novel state-of-the-art for open-vocabulary object detection. Our code is available at https://github.com/xiaofeng94/VL-PLM.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-031-20077-9_10

Learning Semantic Segmentation from Multiple Datasets with Label Shifts

October 24, 2022/in Publications/by NEC Labs America

Publication Date: 10/24/2022

Event: ECCV 2022

Reference: LNCS 13688, pp. 20–36, 2022.

Authors: Dongwan Kim, NEC Laboratories America, Inc., Seoul National University; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Yumin Suh, NEC Laboratories America, Inc.; Masoud Faraki, NEC Laboratories America, Inc.; Sparsh Garg, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Bohyung Ha, Seoul National University

Abstract: While it is desirable to train segmentation models on an aggregation of multiple datasets, a major challenge is that the label space of each dataset may be in conflict with one another. To tackle this challenge, we propose UniSeg, an effective and model-agnostic approach to automatically train segmentation models across multiple datasets with heterogeneous label spaces, without requiring any manual relabeling efforts. Specifically, we introduce two new ideas that account for conflicting and co-occurring labels to achieve better generalization performance in unseen domains. First, we identify a gradient conflict in training incurred by mismatched label spaces and propose a class-independent binary cross-entropy loss to alleviate such label conflicts. Second, we propose a loss function that considers class-relationships across datasets for a better multi-dataset training scheme. Extensive quantitative and qualitative analyses on road-scene datasets show that UniSeg improves over multi-dataset baselines, especially on unseen datasets, e.g., achieving more than 8%p gain in IoU on KITTI. Furthermore, UniSeg achieves 39.4% IoU on the WildDash2 public benchmark, making it one of the strongest submissions in the zero-shot setting. Our project page is available at https://www.nec-labs.com/~mas/UniSeg.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-031-19815-1_2

Media Analytics | Publications

Improving Pseudo Labels for Open-Vocabulary Object Detection

Confidence and Dispersity Speak: Characterizing Prediction Matrix for Unsupervised Accuracy Estimation

Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!

NeurOCS: Neural NOCS Supervision for Monocular 3D Object Localization

Q: How to Specialize Large Vision Language Models to Data Scarce VQA Tasks? A: Self Train on Unlabeled Images!

OmniLabel: A Challenging Benchmark for Language Based Object Detection

Split to Learn: Gradient Split for Multi-Task Human Image Analysis

Single-Stream Multi-level Alignment for Vision-Language Pretraining

Exploiting Unlabeled Data with Vision and Language Models for Object Detection

Learning Semantic Segmentation from Multiple Datasets with Label Shifts

Contact Us

About Us

Our Pages

Read Our Blog Posts