FG-CLIP: Fine-Grained Visual and Textual Alignment

TL;DR

FG-CLIP tackles the challenge of fine-grained visual-textual alignment by combining global-long-caption learning with regional grounding and hard negative sampling. It introduces a two-stage training pipeline and the FineHARD dataset (12M images, 40M region boxes, 10M hard negatives) built from recaptioned data and GRIT-based selections, enabling precise region-to-text correspondences. Empirical results across fine-grained understanding, open-vocabulary detection, and retrieval benchmarks demonstrate significant gains over CLIP and several state-of-the-art methods. The work provides a scalable data-and-methodology blueprint for enhancing fine-grained multimodal reasoning and offers resources to the community for further research and application in multimodal perception and localization.

Abstract

Contrastive Language-Image Pre-training (CLIP) excels in multimodal tasks such as image-text retrieval and zero-shot classification but struggles with fine-grained understanding due to its focus on coarse-grained short captions. To address this, we propose Fine-Grained CLIP (FG-CLIP), which enhances fine-grained understanding through three key innovations. First, we leverage large multimodal models to generate 1.6 billion long caption-image pairs for capturing global-level semantic details. Second, a high-quality dataset is constructed with 12 million images and 40 million region-specific bounding boxes aligned with detailed captions to ensure precise, context-rich representations. Third, 10 million hard fine-grained negative samples are incorporated to improve the model's ability to distinguish subtle semantic differences. We construct a comprehensive dataset, termed FineHARD, by integrating high-quality region-specific annotations with hard fine-grained negative samples. Corresponding training methods are meticulously designed for these data. Extensive experiments demonstrate that FG-CLIP outperforms the original CLIP and other state-of-the-art methods across various downstream tasks, including fine-grained understanding, open-vocabulary object detection, image-text retrieval, and general multimodal benchmarks. These results highlight FG-CLIP's effectiveness in capturing fine-grained image details and improving overall model performance. The data, code, and models are available at https://github.com/360CVGroup/FG-CLIP.

Figures (6)

• Figure 1: Overview of the FG-CLIP. $CLS_{img}$ denotes the image class features output by the Vision Transformer (ViT), while $CLS_{text}$ represents the class features summarized by the text model for multiple inputs, including long captions, short captions, region captions, and positive&negative descriptions of specific regions within images. FG-CLIP's training proceeds in two stages: the first stage leverages global-level caption-image pairs to achieve initial fine-grained alignment, while the second stage supplements these with additional region-level captions, including detailed region captions and positive/negative region descriptions to further refine the alignment.
• Figure 2: Examples of curated visual grounding data.
• Figure 3: Examples of positive and negative descriptions related to image regions.
• Figure 4: Feature visual comparisons of different methods.
• Figure 5: Feature visual comparisons of different input texts.