FILA: Fine-Grained Vision Language Models

TL;DR

The paper tackles fine-grained vision-language understanding under high-resolution imagery, where traditional low-resolution encoders and cropping-based methods fail to preserve global context. It introduces FILA, featuring a Hybrid Encoder that fuses global high-resolution information with sub-image encodings, and the ConvNeXt-ViT Deep Fusion Module (CVFM) for deep multi-level feature integration. Through a two-stage training regime and extensive cross-domain evaluation, the approach achieves state-of-the-art or near-SOTA performance on ten benchmarks, including notable gains on TextVQA and DocVQA. This work demonstrates that integrating dual high-resolution pathways and deep feature fusion significantly enhances fine-grained recognition in multimodal tasks, with broad implications for high-resolution VL understanding and document analysis.

Abstract

Recently, there has been growing interest in the capability of multimodal large language models (MLLMs) to process high-resolution images. A common approach currently involves dynamically cropping the original high-resolution image into smaller sub-images, which are then fed into a vision encoder that was pre-trained on lower-resolution images. However, this cropping approach often truncates objects and connected areas in the original image, causing semantic breaks. To address this limitation, we introduce HyViLM, designed to process images of any resolution while retaining the overall context during encoding. Specifically, we: (i) Design a new visual encoder called Hybrid Encoder that not only encodes individual sub-images but also interacts with detailed global visual features, significantly improving the model's ability to encode high-resolution images. (ii) Propose an optimal feature fusion strategy for the dynamic cropping approach, effectively leveraging information from different layers of the vision encoder. Compared with the state-of-the-art MLLMs under the same setting, our HyViLM outperforms existing MLLMs in nine out of ten tasks. Specifically, HyViLM achieves a 9.6% improvement in performance on the TextVQA task and a 6.9% enhancement on the DocVQA task.

Figures (5)

• Figure 1: (a) An overall performance comparison between FILA and other existing MLLMs. (b) and (c) are examples of image fragments. The solid red line in the figures represents the cropping boundary. LLaVA-NeXT provides correct answers when pivotal information remains within the cropping boundaries. However, even slight overlaps with the boundaries cause it to misidentify the information.
• Figure 2: An Overview of High-Resolution Methods. This Figure illustrates the differences between our method and previous methods.
• Figure 3: Whole architecture for MLLMs. (a) is the first stage of training, during which we only activate the CVFM and projector. (b) is the second stage of training, where we activate all modules of the model. (c) is the structural diagram of Hybrid Encoder, which features high-resolution branches working collaboratively.
• Figure 4: Qualitative Analysis. The issue of image fragmentation has been greatly reduced after incorporating Hybrid Encoder.
• Figure 5: Qualitative results. The red solid line in the image represents the cropping boundary, and the blue box indicates the location of the answer. All images are sized at $672 \times 672$. Neither the red solid line nor the blue box will be input into the model.