Driving Video Retrieval for Complex Queries with Structured Grounding

Publication Date: 5/27/2026

Event: https://arxiv.org/abs/2606.09109

Reference: pp. 1-30

Authors: Manyi Yao, NEC Laboratories America, Inc., University of California, Riverside; Sparsh Garg, NEC Laboratories America, Inc.; Christian Shelton, University of California, Riverside; Amit Roy-Chowdhury, University of California, Riverside; Abhishek Aich, NEC Laboratories America, Inc.

Abstract: Video retrieval at scale is central to data curation and safety validation in autonomous driving, where users want to find not only scenes but also dynamic events such as cut-ins and hard braking. Existing vision-language and keyword-based retrieval methods often miss these events because the relevant motion may not be explicitly described in text or captured by lexical overlap. Rule-based retrieval can encode such events more directly, but it is brittle: generated or hand-written rules often fail when their assumptions do not match real driving data. We propose STRIVE-D, a data-calibrated retrieval framework for driving videos. It uses weakly labeled in-domain videos to estimate when a query rule is reliable, adapt rules that mismatch observed data, and fuse calibrated rule scores with vision-language and keyword-based retrieval signals. Across three driving benchmarks, including newly released human-annotated event data on DrivingDojo, STRIVE-D delivers up to 84% relative improvement in top-1 accuracy over state-of-the-art methods.

Publication Link: https://arxiv.org/abs/2606.09109