Orion: A Unified Visual Agent for Multimodal Perception, Advanced Visual Reasoning and Execution

TL;DR

Orion tackles the limitations of monolithic vision-language models by introducing an agent-based, tool-augmented visual system that plans, executes, and verifies multi-step workflows across images, video, and documents. It fuses large-language-model-style reasoning with a rich library of domain-specific vision tools to achieve precise, production-grade visual intelligence. Across 46 diverse tasks and benchmarks, Orion demonstrates competitive performance with reduced hallucinations and robust, verifiable outputs, enabling complex workflows such as structured data extraction, cross-modal reasoning, and high-fidelity generation. By bridging neural perception with symbolic execution and providing a schema-driven, API-ready interface, Orion advances practical visual AI for real-world applications and enterprise deployment.

Abstract

We introduce Orion, a visual agent that integrates vision-based reasoning with tool-augmented execution to achieve powerful, precise, multi-step visual intelligence across images, video, and documents. Unlike traditional vision-language models that generate descriptive outputs, Orion orchestrates a suite of specialized computer vision tools, including object detection, keypoint localization, panoptic segmentation, Optical Character Recognition (OCR), and geometric analysis, to execute complex multi-step visual workflows. The system achieves competitive performance across MMMU, MMBench, DocVQA, and MMLongBench while extending monolithic VLM capabilities to production-grade visual intelligence. Through its agentic, tool-augmented approach, Orion enables autonomous visual reasoning that bridges neural perception with symbolic execution, marking the transition from passive visual understanding to active, tool-driven visual intelligence. Try Orion for free at: https://chat.vlm.run Learn more at: https://www.vlm.run/orion

Figures (17)

• Figure 1: Orion's Image Understanding, Reasoning, and Execution Capabilities: In the illustration above, we showcase 8 of the many capabilities of Orion - including captioning, detection, segmentation, pointing, tool-calling, image-generation and video-generation – all orchestrated from the original image as the input.
• Figure 2: System architecture of Orion, illustrating the visual agent and its interaction with various visual tools, skills and code execution environments. Orion supports text, image, video and document inputs to answer questions (in multiple steps if possible) that may involve captioning, tagging, detection, generation or external tool-calling. See capabilities \ref{['sec:capabilities']} section for more details.
• Figure 3: Image captioning examples showing dense, rich description generation for scene understanding.
• Figure 4: Visual question answering examples demonstrating open-ended queries with spatial grounding and contextual responses.
• Figure 5: Examples of object, person, and face detection capabilities showing bounding box localization with confidence scores.