Visual-Linguistic Agent: Towards Collaborative Contextual Object Reasoning

TL;DR

This work introduces the Visual-Linguistic Agent (VLA), a collaborative framework that combines the relational reasoning strengths of MLLMs with the precise localization capabilities of traditional object detectors to significantly enhance both spatial reasoning and object localization.

Abstract

Multimodal Large Language Models (MLLMs) excel at descriptive tasks within images but often struggle with precise object localization, a critical element for reliable visual interpretation. In contrast, traditional object detection models provide high localization accuracy but frequently generate detections lacking contextual coherence due to limited modeling of inter-object relationships. To address this fundamental limitation, we introduce the \textbf{Visual-Linguistic Agent (VLA), a collaborative framework that combines the relational reasoning strengths of MLLMs with the precise localization capabilities of traditional object detectors. In the VLA paradigm, the MLLM serves as a central Linguistic Agent, working collaboratively with specialized Vision Agents for object detection and classification. The Linguistic Agent evaluates and refines detections by reasoning over spatial and contextual relationships among objects, while the classification Vision Agent offers corrective feedback to improve classification accuracy. This collaborative approach enables VLA to significantly enhance both spatial reasoning and object localization, addressing key challenges in multimodal understanding. Extensive evaluations on the COCO dataset demonstrate substantial performance improvements across multiple detection models, highlighting VLA's potential to set a new benchmark in accurate and contextually coherent object detection.

Figures (3)

• Figure 1: Comparison of traditional object detection (a) and MLLM (b) approaches. Traditional object detection focuses on reasoning among local objects without global context, limiting its ability to model object relationships. In contrast, MLLMs incorporate both local and global context, enabling more comprehensive object relationship modeling.
• Figure 2: The proposed Visual-Linguistic-Agent (VLA) paradigm. The Visual Agent (e.g., YOLO) detects objects and generates bounding boxes with class labels, which are passed to the Linguistic Agent (MLLM) for reasoning and contextual analysis. Based on the MLLM's assessment, false detections are filtered, and the Classification Visual Agent corrects erroneous detections. This collaboration between agents enhances object detection accuracy and provides more contextually coherent results.
• Figure 3: Examples of error correction by VLA using DINO with MLLMs as the Linguistic Agent. The figure highlights common misclassifications corrected by VLA, such as identifying a dog instead of a horse, recognizing a kite instead of an umbrella, and correctly labeling a cow instead of a sheep. These corrections demonstrate VLA's ability to refine object detection outputs by leveraging linguistic reasoning to resolve ambiguous or erroneous labels.