OVOD-Agent: A Markov-Bandit Framework for Proactive Visual Reasoning and Self-Evolving Detection

TL;DR

OVOD-Agent tackles open-vocabulary object detection by replacing static category matching with proactive, interpretable Visual-CoT reasoning grounded in a discrete eight-state weak Markovian framework. A UCB-based Bandit guides action-driven exploration, while a lightweight Reward Model distills both transition dynamics and weak rewards to enable self-evolving detection without large LLMs. The method yields consistent gains on LVIS and COCO, notably improving rare-category AP with minimal latency overhead, and demonstrates robust cross-backbone applicability. This approach provides a scalable, deployable pathway for open-world detection, combining principled uncertainty-driven exploration with structured, interpretable state transitions. Practically, OVOD-Agent can augment existing OVOD backbones with modest memory and computation, making enhanced open-vocabulary grounding feasible in real-world applications.

Abstract

Open-Vocabulary Object Detection (OVOD) aims to enable detectors to generalize across categories by leveraging semantic information. Although existing methods are pretrained on large vision-language datasets, their inference is still limited to fixed category names, creating a gap between multimodal training and unimodal inference. Previous work has shown that improving textual representation can significantly enhance OVOD performance, indicating that the textual space is still underexplored. To this end, we propose OVOD-Agent, which transforms passive category matching into proactive visual reasoning and self-evolving detection. Inspired by the Chain-of-Thought (CoT) paradigm, OVOD-Agent extends the textual optimization process into an interpretable Visual-CoT with explicit actions. OVOD's lightweight nature makes LLM-based management unsuitable; instead, we model visual context transitions as a Weakly Markovian Decision Process (w-MDP) over eight state spaces, which naturally represents the agent's state, memory, and interaction dynamics. A Bandit module generates exploration signals under limited supervision, helping the agent focus on uncertain regions and adapt its detection policy. We further integrate Markov transition matrices with Bandit trajectories for self-supervised Reward Model (RM) optimization, forming a closed loop from Bandit exploration to RM learning. Experiments on COCO and LVIS show that OVOD-Agent provides consistent improvements across OVOD backbones, particularly on rare categories, confirming the effectiveness of the proposed framework.

Figures (5)

• Figure 1: We illustrate the state-transition behavior of OVOD-Agent as it iteratively updates its category hypothesis. Starting from an initial dictionary lookup, the agent applies attribute-aware actions that adjust color, texture, and spatial cues to produce a more accurate and grounded state description. The number of required actions varies across images, from single-step updates to multi-step reasoning.
• Figure 2: OVOD-Agent operates through a self-evolving visual reasoning pipeline. (a) The environment updates the visual state by applying detector feedback and the current prompt-conditioned context. (b) A UCB-based Bandit module selects and executes visual actions, collecting rewards and empirical transitions from sampled trajectories. (c) The collected trajectories are joined into an image-specific Markov transition matrix that models how weak Markov units evolve under different visual operations and serves as a structured prior for learning. (d) A lightweight Reward--Policy Model (RM) is trained on these trajectories and transition priors, distilling both transition behavior and weak reward signals. During inference, the RM replaces Bandit exploration and directly guides the agent's step-by-step refinement process without relying on LLMs.
• Figure 3: Failure cases of OVOD-Agent. Representative examples where the agent fails to correctly identify rare or occluded objects.
• Figure 4: Step-by-step Case Study of OVOD-Agent, showing how visual actions (color, texture, container, background, spatial cues) progressively refine the caption and stabilize detector grounding.
• Figure 5: Evaluation protocol for GPT-5 trajectory scoring, including the instruction prompt defining the evaluator's role and the input prompt containing the sampled trajectory.