GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs

TL;DR

GranViT tackles the limitation of global-only visual representations in MLLMs by introducing a fine-grained vision encoder trained on Gran-29M, a large-scale dataset with region-level annotations. It couples a two-stage pretraining-adaptation pipeline with region-level autoregressive supervision, using $Bbox2Caption$ for fine-grained feature extraction and $Caption2Bbox$ for LLM-oriented localization, complemented by self-distillation to reinforce local features. Empirical results show GranViT achieving state-of-the-art performance on fine-grained recognition and OCR tasks, with strong transferability to larger LLMs and competitive multimodal VQA and reasoning. The approach provides a scalable path to incorporate precise regional understanding into MLLMs, improving localization, grounding, and overall multimodal performance in diverse tasks.

Abstract

Vision encoders are indispensable for allowing impressive performance of Multi-modal Large Language Models (MLLMs) in vision language tasks such as visual question answering and reasoning. However, existing vision encoders focus on global image representations but overlook fine-grained regional analysis. They are limited in fine grained perception due to the scarcity of fine grained annotated data and the lack of a fine grained pre-training paradigm. In this paper, we propose GranViT, a novel Vision Transformer that integrates fine-grained feature extraction with semantic alignment to Large Language Models (LLMs) via region level autoregressive training. We first construct Gran-29M, a dataset comprising 2million natural and OCR images paired with over 180 million high-quality region-level annotations, to enable large scale fine grained pretraining. Consequently, we develop a pretraining-adaptation framework along with a self distillation mechanism to train fine-grained GranViT on Gran-29M. We sufficiently exploit the fine-grained annotations from Gran-29M to resort to bounding-box-to-caption regression to enhance localized visual representation of the vision encoder in the pretraining and caption-to-bounding-box regression to improve vision feature utilization and localization for LLM in the adaptation. We further incorporate a self distillation mechanism that imposes explicit localization constraints on the vision encoder to strengthen its regional reasoning capability. Extensive experiments show that GranViT surpasses existing vision encoders and attains strong transferability to varying LLMs. Remarkably, it achieves state-of-the-art results on fine-grained recognition, multimodal VQA, and OCR understanding.

Figures (7)

• Figure 1: (a) Compared to existing vision encoders, GranViT demonstrates outstanding performance across fine-grained natural image and OCR understanding. HBench denotes HallusionBench. (b) Attention visualization of existing vision encoders according to the query token. The small red rectangle indicates the query token. Best viewed with zoom in.
• Figure 2: The details of data annotations of Gran-29M. We leverage ViTDet li2022exploring and Qwen2.5-VL-7B bai2025qwen2 for bbox and caption generation. Then, we transfer the absolute bbox coordinate to a relative one based on the image resolution and apply rigorous filtering based on image resolution, bbox area, and the number of bboxes per image. Finally, we reformat the global and local captions into QA pairs.
• Figure 3: The fine-grained pretraining and transferring paradigm of GranViT. For pretraining, the vision encoder and projector are tuned via the global and $Bbox2Caption$ task for fine-grained feature extraction. The teacher vision encoder explicitly supervises the local region of features extracted by the student vision encoder. For vision feature adaptation and transfer, based on the fine-grained vision encoder, we apply LLM tuning to further strengthen the localization capability of the LLM regarding fine-grained visual features via the global and $Caption2Bbox$ task.
• Figure 4: (a) The performance curve of Stage1 and Stage2. We sample 8M $Bbox2Caption$ and $Caption2Bbox$ samples respectively for pretraining and adaptation and calculate ROUGE-L barbella2022rouge and ACC@IOU0.5 for $Bbox2Caption$ and $Caption2Bbox$ respectively. In stage 1, the ACC@IOU0.5 of the $Caption2Bbox$ task only achieves $13\%$, while the ROUGE-L of the $Bbox2Caption$ task achieves $52\%$. Conversely, in stage 2, the training of LLM leads to a notable increase in ACC@IOU0.5 for $Caption2Bbox$, while $Bbox2Caption$ achieves only a minimal improvement of $3\%$. (b) Visualization of predicted bbox coordinate of $Caption2Bbox$ task in stage 1 and stage 2. Green bboxes indicate predicted regions, while red ones denote ground truth.
• Figure 5: Visualization of Gran-29M.