VIRO: Robust and Efficient Neuro-Symbolic Reasoning with Verification for Referring Expression Comprehension

TL;DR

This work tackles Referring Expression Comprehension with frequent no-target scenarios by introducing VIRO, a neuro-symbolic framework that embeds lightweight, operator-level verifications into each reasoning step. A two-stage pipeline translates queries into verifiable symbolic programs and executes them with per-operator checks (uncertainty verification via CLIP and logical verification for spatial relations), enabling explicit no-target abstention and reducing error cascades. VIRO achieves state-of-the-art robustness in zero-target settings (61.1% Balanced Accuracy) while maintaining high throughput and a program-failure rate below 0.3%, and it generalizes to real-world egocentric data (RefEgo) with scalable 1-to-N inference via decoupled program generation. The approach demonstrates the practical impact of coupling verification with neuro-symbolic reasoning for trustworthy multimodal grounding, and its modular design invites extensions to interactive and embodied AI scenarios. Overall, VIRO delivers robust, efficient, and scalable REC with explicit no-target handling, advancing the reliability of visual grounding in diverse real-world contexts.

Abstract

Referring Expression Comprehension (REC) aims to localize the image region corresponding to a natural-language query. Recent neuro-symbolic REC approaches leverage large language models (LLMs) and vision-language models (VLMs) to perform compositional reasoning, decomposing queries 4 structured programs and executing them step-by-step. While such approaches achieve interpretable reasoning and strong zero-shot generalization, they assume that intermediate reasoning steps are accurate. However, this assumption causes cascading errors: false detections and invalid relations propagate through the reasoning chain, yielding high-confidence false positives even when no target is present in the image. To address this limitation, we introduce Verification-Integrated Reasoning Operators (VIRO), a neuro-symbolic framework that embeds lightweight operator-level verifiers within reasoning steps. Each operator executes and validates its output, such as object existence or spatial relationship, thereby allowing the system to robustly handle no-target cases when verification conditions are not met. Our framework achieves state-of-the-art performance, reaching 61.1% balanced accuracy across target-present and no-target settings, and demonstrates generalization to real-world egocentric data. Furthermore, VIRO shows superior computational efficiency in terms of throughput, high reliability with a program failure rate of less than 0.3%, and scalability through decoupled program generation from execution.

Figures (10)

• Figure 1: Missing verification leads to forced predictions, which VIRO prevents. Illustrative comparison between previous REC methods and our VIRO framework in no-target cases. Previous REC methods (left) are forced to output a prediction, even when the query cannot be grounded in the image, due to the lack of a mechanism for eliminating incorrect candidates. In contrast, our VIRO framework (right) terminates early instead of hallucinating a prediction: (i) FIND operator identifies that there is no elephant in the image (top); (ii) FIND_DIRECTION operator identifies the person is not positioned to the left of the elephant (bottom).
• Figure 2: VIRO vs. prior compositional REC. (i) VIRO generates a symbolic program with an LLM, then an interpreter executes operators with operator-level verification; as exemplified by the FIND operator using CLIP-based uncertainty verification to prune OVD proposals. In contrast, (ii) ViperGPT compiles queries into Python code but lacks verification, while (iii) HYDRA and (iv) NAVER tightly couple program generation and execution, requiring iterative generators for all inputs.
• Figure 3: Total inference time (s) in a 1-query-$N$-images setting, with the $x$-axis on a logarithmic scale.
• Figure 4: Analysis of the OVD detection threshold, illustrating the trade-off between TPR and TNR.
• Figure 5: Ablation study of the proposed verification components in VIRO. 'Fixed' refers to a fixed threshold, while 'adaptive' refers to an adaptive threshold.