PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learning

TL;DR

PixCLIP tackles the challenge of fine-grained visual-language grounding by enabling pixel-level alignment between arbitrary image regions and long textual descriptions. It introduces LongGRIT, a 1.5 million sample dataset of pixel-level region captions, and a three-branch framework comprising global mask-text contrastive alignment, fine-grained cropping alignment, and local-global representation enhancement, using an LLM-based text encoder and a modified vision encoder with mask-aware patch embeddings. The model achieves state-of-the-art results on region-level tasks and image-text retrieval, including long-text scenarios, demonstrating robust regional focus and long-form text handling. This work broadens CLIP-like models toward universal, fine-grained grounding and provides a scalable data pipeline for pixel-text alignment benchmarks.

Abstract

While the Contrastive Language-Image Pretraining(CLIP) model has achieved remarkable success in a variety of downstream vison language understanding tasks, enhancing its capability for fine-grained image-text alignment remains an active research focus. To this end, most existing works adopt the strategy of explicitly increasing the granularity of visual information processing, e.g., incorporating visual prompts to guide the model focus on specific local regions within the image. Meanwhile, researches on Multimodal Large Language Models(MLLMs) have demonstrated that training with long and detailed textual descriptions can effectively improve the model's fine-grained vision-language alignment. However, the inherent token length limitation of CLIP's text encoder fundamentally limits CLIP to process more granular textual information embedded in long text sequences. To synergistically leverage the advantages of enhancing both visual and textual content processing granularity, we propose PixCLIP, a novel framework designed to concurrently accommodate visual prompt inputs and process lengthy textual descriptions. Specifically, we first establish an automated annotation pipeline capable of generating pixel-level localized, long-form textual descriptions for images. Utilizing this pipeline, we construct LongGRIT, a high-quality dataset comprising nearly 1.5 million samples. Secondly, we replace CLIP's original text encoder with the LLM and propose a three-branch pixel-text alignment learning framework, facilitating fine-grained alignment between image regions and corresponding textual descriptions at arbitrary granularity. Experiments demonstrate that PixCLIP showcases breakthroughs in pixel-level interaction and handling long-form texts, achieving state-of-the-art performance.

Figures (11)

• Figure 1: As shown on the (a), prior methods fail on fine-grained local tasks due to inability to accept mask input, While existing promptable methods struggle to handle long texts due to token limitations, as shown in (b). Only our method succeeds in yielding robust embeddings for image and text at any granularity.
• Figure 2: The framework of the proposed PixCLIP. Our model structure allows for inheriting weights from existing model and we propose the Fine-grained Cropping Alignment and Local-Global Representation Enhancement branches to enhance the mask-based visual embeddings.
• Figure 3: The pipeline of our data generation method. Using multiple MLLMs, we generate object captions and in-context captions. They are then checked and merged into fine-grained captions for use in further training stage.
• Figure 4: Retrieval comparison with previous models in traditional Image-Text benchmarks.
• Figure 5: Performance comparison between PixCLIP and prior state-of-the-art works.