MM TTA: Multi Modal Test Time Adaptation for 3D Semantic Segmentation

/in /by

Publication Date: 4/22/2022

Event: arXiv

Reference: https://arxiv.org/abs/2204.12667

Authors: Inkyu Shin, KAIST, NEC Laboratories America, Inc., Yi Hsuan Tsai, NEC Laboratories America, Inc., Bingbing Zhuang, NEC Laboratories America, Inc., Samuel Schulter, NEC Laboratories America, Inc., Buyu Liu, NEC Laboratories America, Inc., Sparsh Garg, NEC Laboratories America, Inc., In So Kweon, KAIST, Kuk Jin Yoon, KAIST

Abstract: Test time adaptation approaches have recently emerged as a practical solution for handling domain shift without access to the source domain data. In this paper, we propose and explore a new multi modal extension of test time adaptation for 3D semantic segmentation. We find that directly applying existing methods usually results in performance instability at test time because multi modal input is not considered jointly. To design a framework that can take full advantage of multi modality, where each modality provides regularized self supervisory signals to other modalities, we propose two complementary modules within and across the modalities. First, Intra modal Pseudolabel Generation (Intra PG) is introduced to obtain reliable pseudo labels within each modality by aggregating information from two models that are both pre trained on source data but updated with target data at different paces. Second, Inter modal Pseudo label Refinement (Inter PR) adaptively selects more reliable pseudo labels from different modalities based on a proposed consistency scheme. Experiments demonstrate that our regularized pseudo labels produce stable self learning signals in numerous multi modal test time adaptation scenarios for 3D semantic segmentation. Visit our project website at https://www.nec labs.com/˜mas/MM TTA

Publication Link: https://arxiv.org/pdf/2204.12667.pdf

You might also like
5 Accepted Papers CVPR2022! Check out our Computervision Papers at #CVPR2022
5 Accepted Papers CVPR2022! Active Adversarial Domain Adaptation
5 Accepted Papers CVPR2022! Adaptation Across Extreme Variations using Unlabeled Bridges
5 Accepted Papers CVPR2022! Adversarial Alignment for Source Free Object Detection
5 Accepted Papers CVPR2022! Adversarial Learning of Privacy-Preserving and Task-Oriented Representations
5 Accepted Papers CVPR2022! Channel Recurrent Attention Networks for Video Pedestrian Retrieval
5 Accepted Papers CVPR2022! Controllable Dynamic Multi-Task Architectures
5 Accepted Papers CVPR2022! Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zero-shot Classification and Retrieval of Videos