Learning Semantic Segmentation from Multiple Datasets with Label Shifts

/in /by

Publication Date: 10/24/2022

Event: ECCV 2022

Reference: LNCS 13688, pp. 20–36, 2022.

Authors: Dongwan Kim, NEC Laboratories America, Inc., Seoul National University; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Yumin Suh, NEC Laboratories America, Inc.; Masoud Faraki, NEC Laboratories America, Inc.; Sparsh Garg, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Bohyung Han, Seoul National University

Abstract: While it is desirable to train segmentation models on an aggregation of multiple datasets, a major challenge is that the label space of each dataset may be in conflict with one another. To tackle this challenge, we propose UniSeg, an effective and model-agnostic approach to automatically train segmentation models across multiple datasets with heterogeneous label spaces, without requiring any manual relabeling efforts. Specifically, we introduce two new ideas that account for conflicting and co-occurring labels to achieve better generalization performance in unseen domains. First, we identify a gradient conflict in training incurred by mismatched label spaces and propose a class-independent binary cross-entropy loss to alleviate such label conflicts. Second, we propose a loss function that considers class-relationships across datasets for a better multi-dataset training scheme. Extensive quantitative and qualitative analyses on road-scene datasets show that UniSeg improves over multi-dataset baselines, especially on unseen datasets, e.g., achieving more than 8%p gain in IoU on KITTI. Furthermore, UniSeg achieves 39.4% IoU on the WildDash2 public benchmark, making it one of the strongest submissions in the zero-shot setting. Our project page is available at https://www.nec-labs.com/~mas/UniSeg.

Publication Link: https://link.springer.com/chapter/10.1007/978-3-031-19815-1_2

Additional Publication Link: https://arxiv.org/pdf/2202.14030.pdf