Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering

/in /by

Publication Date: 8/3/2026

Event: https://arxiv.org

Reference: https://arxiv.org/pdf/2608.02907

Authors: Jingchen Sun, State University of New York at Buffalo; Shaobo Han, NEC Laboratories America, Inc.; Ruiyi Zhang, Adobe Research; Naresh Kumar Devulapally, State University of New York at Buffalo; Ming Liu, Iowa State University; Yitao Long, New York University; Vishnu Suresh Lokhande, State University of New York at Buffalo; Changyou Chen, State University of New York at Buffalo

Abstract: Multimodal retrievers are essential for knowledge-based visual question answering, where they retrieve external evidence for image-question pairs. However, existing contrastive training methods typically treat all unmatched query-document pairs as equally informative negatives, which is problematic because many unmatched documents may still be semantically relevant or partially useful. We propose Bayesian Data Reweighting, a probabilistic framework that models query-document importance as latent variables and adaptively infers posterior weights to downweight likely false negatives. With closed-form posterior updates under conjugate priors and stochastic EM optimization, our method consistently improves retrieval accuracy across three retrievers and seven knowledge-based VQA benchmarks.

Publication Link: https://arxiv.org/pdf/2608.02907