MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation

Publication Date: 6/19/2022

Event: CVPR’22

Reference: pp. 16928-16937, 2022

Authors: Inkyu Shin, NEC Laboratories America, Inc., KAIST; Yi-Hsuan Tsai, NEC Laboratories America, Inc.; Bingbing Zhuang, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Buyu Liu, NEC Laboratories America, Inc.; Sparsh Garg, NEC Laboratories America, Inc.; In So Kweon, KAIST; Kuk-Jin Yoon, KAIST

Abstract: Test-time adaptation approaches have recently emerged as a practical solution for handling domain shift without access to the source domain data. In this paper, we propose and explore a new multi-modal extension of test-time adaptation for 3D semantic segmentation. We find that, directly applying existing methods usually results in performance instability at test time, because multi-modal input is not considered jointly. To design a framework that can take full advantage of multi-modality, where each modality provides regularized self-supervisory signals to other modalities, we propose two complementary modules within and across the modalities. First, Intra-modal Pseudo-label Generation (Intra-PG) is introduced to obtain reliable pseudo labels within each modality by aggregating information from two models that are both pre-trained on source data but updated with target data at different paces. Second, Inter-modal Pseudo-label Refinement (Inter-PR) adaptively selects more reliable pseudo labels from different modalities based on a proposed consistency scheme. Experiments demonstrate that our regularized pseudo labels produce stable self-learning signals in numerous multi-modal test-time adaptation scenarios for 3D semantic segmentation. Visit our project website at https://www.nec-labs.com/~mas/MM-TTA

Publication Link: https://ieeexplore.ieee.org/document/9879729

Additional Publication Link: https://arxiv.org/pdf/2204.12667.pdf