Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models

TL;DR

This work introduces a novel fine-grained visual knowledge alignment method that effectively aligns and integrates multi-scale knowledge of objects, including texts, coordinates, and images, which is underpinned by the multi-scale fine-grained enhancement data synthesis pipeline.

Abstract

Multi-modal large language models (MLLMs) have achieved remarkable success in fine-grained visual understanding across a range of tasks. However, they often encounter significant challenges due to inadequate alignment for fine-grained knowledge, which restricts their ability to accurately capture local details and attain a comprehensive global perception. While recent advancements have focused on aligning object expressions with grounding information, they typically lack explicit integration of object images, which contain affluent information beyond mere texts or coordinates. To bridge this gap, we introduce a novel fine-grained visual knowledge alignment method that effectively aligns and integrates multi-scale knowledge of objects, including texts, coordinates, and images. This innovative method is underpinned by our multi-scale fine-grained enhancement data synthesis pipeline, which provides over 300K essential training data to enhance alignment and improve overall performance. Furthermore, we present TinyGroundingGPT, a series of compact models optimized for high-level alignments. With a scale of approximately 3B parameters, TinyGroundingGPT achieves outstanding results in grounding tasks while delivering performance comparable to larger MLLMs in complex visual scenarios.

Figures (15)

• Figure 1: The comparison of alignment for multi-scale object representations. The C, T, I denote object coordinates, texts and images respectively. The "X-Y" denote MLLMs handle input "X" and output "Y".
• Figure 2: Illustration of the proposed multi-modal fine-grained visual knowledge alignment method. It adopts a three-stage training strategy that progresses from easy to hard and the multi-scale fine-grained enhancement data synthesis pipeline constructs over 300K fine-grained alignment data.
• Figure 3: The model architecture of our proposed TinyGroundingGPT. It utilizes multi-scale visual encoders and supports queries regarding different object representations. Object images are cropped and zoomed from the input image according to the input coordinates.
• Figure 4: A case for the outputs of our TinyGroundingGPT when the input is either enhanced with coordinates or not. Probability values indicate the likelihood of generating corresponding tokens.
• Figure 5: Visualization of the attention map for image patches with different object representation outputs (texts, coordinates, and images, underlined). The red bounding box denotes the target region. The attention at the four corners serves as anchors for grounding, while attention at specific objects highlights their importance.