Exploring Question Decomposition for Zero-Shot VQA

Publication Date: 12/11/2023

Event: NeurIPS 2023

Reference: pp. 1-13, 2023

Authors: Zaid Khan, Northeastern University, NEC Laboratories America, Inc.; Vijay Kumar B G, NEC Laboratories America, Inc.; Samuel Schulter, NEC Laboratories America, Inc.; Manmohan Chandraker, NEC Laboratories America, Inc., UC San Diego; Yun Fu, Northeastern University

Abstract: Visual question answering (VQA) has traditionally been treated as a single-step task where each question receives the same amount of effort, unlike natural human question-answering strategies. We explore a question decomposition strategy for VQA to overcome this limitation. We probe the ability of recently developed large vision-language models to use human-written decompositions and produce their own decompositions of visual questions, finding they are capable of learning both tasks from demonstrations alone. However, we show that naive application of model-written decompositions can hurt performance. We introduce a model-driven selective decomposition approach for second-guessing predictions and correcting errors, and validate its effectiveness on eight VQA tasks across three domains, showing consistent improvements in accuracy, including improvements of >20% on medical VQA datasets and boosting the zero-shot performance of BLIP-2 above chance on a VQA reformulation of the challenging Winoground task. Project Site: https://zaidkhan.me/decomposition-0shot-vqa/

Publication Link: https://proceedings.neurips.cc/paper_files/paper/2023/file/b14cf0a01f7a8b9cd3e365e40f910272-Paper-Conference.pdf