Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing

TL;DR

Vitron introduces a unified pixel-level vision LLM that comprehends, generates, segments, and edits both images and videos by coupling frontend encoders, a central LLM, and backend visual specialists. It employs a novel hybrid message-passing scheme that blends discrete textual instructions with continuous signal embeddings to drive downstream modules, alongside pixel-level spatiotemporal alignment and a cross-task synergy module. Across 12 visual tasks and 22 datasets, Vitron achieves competitive or superior performance to specialist models, while enabling flexible generation and editing workflows. This work advances toward a true multimodal generalist with unified task support and cross-task knowledge sharing, promising broader practical impact in vision-language AI.

Abstract

Recent developments of vision large language models (LLMs) have seen remarkable progress, yet still encounter challenges towards multimodal generalists, such as coarse-grained instance-level understanding, lack of unified support for both images and videos, and insufficient coverage across various vision tasks. In this paper, we present VITRON, a universal pixel-level vision LLM designed for comprehensive understanding, generating, segmenting, and editing of both static images and dynamic videos. Building on top of an LLM backbone, VITRON incorporates encoders for images, videos, and pixel-level regional visuals within its frontend modules, while employing state-of-the-art visual specialists as its backend, via which VITRON supports a spectrum of vision end tasks, spanning visual comprehension to visual generation, from low level to high level. To ensure an effective and precise message passing from LLM to backend modules for function invocation, we propose a novel hybrid method by simultaneously integrating discrete textual instructions and continuous signal embeddings. Further, we design various pixel-level spatiotemporal vision-language alignment learning for VITRON to reach the best fine-grained visual capability. Finally, a cross-task synergy module is advised to learn to maximize the task-invariant fine-grained visual features, enhancing the synergy between different visual tasks. Demonstrated over 12 visual tasks and evaluated across 22 datasets, VITRON showcases its extensive capabilities in the four main vision task clusters. Overall, this work illuminates the great potential of developing a more unified multimodal generalist. Project homepage: https://vitron-llm.github.io/

Figures (13)

• Figure 1: Vitron supports four main task clusters of visions, spanning visual comprehension to visual generation, from low level to high level.
• Figure 2: Technical overview of the Vitron framework.
• Figure 3: Illustration of the synergy module.
• Figure 4: The influences of using different strategies for message passing.
• Figure 5: The impact of various fine-grained visual grounding learning strategies.