FLAIR: VLM with Fine-grained Language-informed Image Representations

TL;DR

FLAIR tackles the challenge of fine-grained vision-language alignment by learning token-level, language-informed image representations through text conditioned attention pooling and diverse sub-caption sampling from long synthetic captions. By training on $30{,}000{,}000$ captioned images, it achieves state-of-the-art performance on standard, fine-grained, and long-caption multimodal retrieval, and delivers substantial gains in zero-shot semantic segmentation. The method introduces two losses, $L^{\text{tcs}}$ and $L^{\text{mps}}$, and a diverse sampling strategy that enables robust global and local alignment without requiring changes to the text encoder. While competitive with large-scale CLIP models on retrieval and segmentation, FLAIR highlights the value of high-quality language-conditioned supervision and suggests that scaling data and parameters can close remaining gaps in zero-shot classification, offering a practical path for fine-grained VLMs in real-world applications.

Abstract

CLIP has shown impressive results in aligning images and texts at scale. However, its ability to capture detailed visual features remains limited because CLIP matches images and texts at a global level. To address this issue, we propose FLAIR, Fine-grained Language-informed Image Representations, an approach that utilizes long and detailed image descriptions to learn localized image embeddings. By sampling diverse sub-captions that describe fine-grained details about an image, we train our vision-language model to produce not only global embeddings but also text-specific image representations. Our model introduces text-conditioned attention pooling on top of local image tokens to produce fine-grained image representations that excel at retrieving detailed image content. We achieve state-of-the-art performance on both, existing multimodal retrieval benchmarks, as well as, our newly introduced fine-grained retrieval task which evaluates vision-language models' ability to retrieve partial image content. Furthermore, our experiments demonstrate the effectiveness of FLAIR trained on 30M image-text pairs in capturing fine-grained visual information, including zero-shot semantic segmentation, outperforming models trained on billions of pairs. Code is available at https://github.com/ExplainableML/flair .

Figures (11)

• Figure 1: Visualization of the similarity scores between local image tokens and different text queries. While previous works zheng2025dreamlipradford2021learning lack fine-grained alignment, FLAIR matches text and image semantics at the token level.
• Figure 2: Comparison of text-conditioned attention pooling with previous methods. (a) Vanilla CLIP ($\mathcal{L}^{\text{clip}}$) aligns global image $\mathbf{v}^{\text{g}}$ and text $\mathbf{t}^{\text{g}}$ tokens. (b) SigLIP ($\mathcal{L}^{\text{sigmoid}}$) employs a global learnable image token $\mathbf{v}^{\text{g}}$ as query for a cross-attention to pool the local tokens $\mathbf{v}_{\text{loc}}$. (c) FLAIR ($\mathcal{L}^{\text{tcs}}$) employs text-conditioned attention pooling that leverages $\mathbf{t}^{\text{g}}$ as query, aggregating $\mathbf{v}^{\text{loc}}$ to capture language-informed visual features. (d) FLAIR ($\mathcal{L}^{\text{tcs}} + \mathcal{L}^{\text{mps}}$) adds an extra multi-positive global sigmoid loss to refine global-level image-text alignment.
• Figure 3: Overview of FLAIR; We sample diverse positive and negative captions $\{T_{1_1}...T_{3_1}\}$ for an image $I_1$. $f_{\text{txt}}$ and $f_{\text{img}}$ then produce the global text tokens $\{ \mathbf{t}^{\text{g}}_{1_1} ... \mathbf{t}^{\text{g}}_{3_1} \}$, the global image token $\mathbf{v}^{\text{g}}_1$, and local image tokens $\mathbf{v}^{\text{loc}}_1$. Conditioned on $\{ \mathbf{t}^{\text{g}}_{1_1} ... \mathbf{t}^{\text{g}}_{3_1} \}$, $f_{\text{AttnPool}(.)}$ generates fine-grained text-conditioned image representations $\{ \mathbf{v}^{\text{tc}}_{1_1} ... \mathbf{v}^{\text{tc}}_{3_1} \}$. The text-conditioned sigmoid loss $\mathcal{L}^{\text{tcs}}$ aligns $\{ \mathbf{t}^{\text{g}}_{1_1} ... \mathbf{t}^{\text{g}}_{3_1} \}$ with $\{ \mathbf{v}^{\text{tc}}_{1_1} ... \mathbf{v}^{\text{tc}}_{3_1} \}$ contrastively, while the multi-positive sigmoid loss $\mathcal{L}^{\text{mps}}$ refines the global alignment between $\mathbf{v}^{\text{g}}_1$ and $\{ \mathbf{t}^{\text{g}}_{1_1} ... \mathbf{t}^{\text{g}}_{3_1} \}$.
• Figure 4: Visualization of attention maps in the attention pooling layer $f_{\text{AttnPool}}(.)$. Regions of high attention are highlighted in red.
• Figure 5: Visualization of the attention maps w.r.t. fine-grained captions. In the images, regions with high attention scores are marked in red; in the captions, objects representing the main semantics of the sentences are marked in red, while objects with less semantic significance are underlined.