Exploiting Unlabeled Data with Vision and Language Models for Object Detection

/in /by

Publication Date: 10/24/2022

Event: ECCV 2022

Reference: LNCS 13669, pp. 159–175, 2022

Authors: Shiyu Zhao, Rutgers University, NEC Laboratories America, Inc.; Zhixing Zhang, Rutgers University, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Long Zhao, Google Research; Vijay Kumar BG, NEC Laboratories America, Inc.; Anastasis Stathopoulos, Rutgers University; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Dimitris Metaxas, Rutgers University

Abstract: Building robust and generic object detection frameworks requires scaling to larger label spaces and bigger training datasets. However, it is prohibitively costly to acquire annotations for thousands of categories at a large scale. We propose a novel method that leverages the rich semantics available in recent vision and language models to localize and classify objects in unlabeled images, effectively generating pseudo labels for object detection. Starting with a generic and class-agnostic region proposal mechanism, we use vision and language models to categorize each region of an image into any object category that is required for downstream tasks. We demonstrate the value of the generated pseudo labels in two specific tasks, open-vocabulary detection, where a model needs to generalize to unseen object categories, and semi-supervised object detection, where additional unlabeled images can be used to improve the model. Our empirical evaluation shows the effectiveness of the pseudo labels in both tasks, where we outperform competitive baselines and achieve a novel state-of-the-art for open-vocabulary object detection. Our code is available at https://github.com/xiaofeng94/VL-PLM.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-031-20077-9_10

Additional Publication Link: https://arxiv.org/pdf/2207.08954.pdf