Aligning Forest and Trees in Images and Long Captions for Visually Grounded Understanding

TL;DR

CAFT addresses the challenge of aligning images with long captions by introducing a hierarchical cross-domain framework that mirrors forest–tree structures in both vision and language. It combines a fine-to-coarse visual encoder with a hierarchical text transformer, trained with part-level ($L^{\text{part}}$) and whole-level ($L^{\text{whole}}$) alignment losses to ground mid-level semantics in sub-captions and top-level semantics in full captions, respectively. Trained on 30M image–text pairs from DreamLIP, CAFT achieves state-of-the-art performance on six long-caption retrieval benchmarks and exhibits strong unsupervised visual grounding, including zero-shot referring segmentation. CAFT++ further improves reliability by blending part- and whole-level similarities during inference, highlighting the practical impact of hierarchical cross-domain grounding for scalable, compositional vision–language understanding. $L^{\text{part}}$ and $L^{\text{whole}}$ losses enable robust grounding, ensuring that global semantics emerge from grounded local evidence rather than opaque aggregation.

Abstract

Large vision-language models such as CLIP struggle with long captions because they align images and texts as undifferentiated wholes. Fine-grained vision-language understanding requires hierarchical semantics capturing both global context and localized details across visual and textual domains. Yet linguistic hierarchies from syntax or semantics rarely match visual organization, and purely visual hierarchies tend to fragment scenes into appearance-driven parts without semantic focus. We propose CAFT (Cross-domain Alignment of Forests and Trees), a hierarchical image-text representation learning framework that aligns global and local semantics across images and long captions without pixel-level supervision. Coupling a fine-to-coarse visual encoder with a hierarchical text transformer, it uses a hierarchical alignment loss that matches whole images with whole captions while biasing region-sentence correspondences, so that coarse semantics are built from fine-grained evidence rather than from aggregation untethered to part-level grounding. Trained on 30M image-text pairs, CAFT achieves state-of-the-art performance on six long-text retrieval benchmarks and exhibits strong scaling behavior. Experiments show that hierarchical cross-domain alignment enables fine-grained, visually grounded image-text representations to emerge without explicit region-level supervision.

Figures (8)

• Figure 1: Long-text image retrieval demands grounded understanding. Identifying the correct image (bottom) requires recognizing both its coarse composition ( what major elements are present: bus, car, building etc.) and its fine-grained details ( what each element is: a modern red double-decker bus labeled route 38). In the wrong image (top), a long caption causes the baseline model to latch onto some dominant descriptions while neglecting others, leading to errors in both scene composition and local details. This example illustrates the need for models that align visual and textual structure across scales, seeing both the forest and the trees.
• Figure 2: Vision-language understanding requires hierarchical semantics to be aligned across domains.Traditional textual hierarchies, such as syntactic parse trees or semantic graphs, do not naturally correspond to the spatial organization of visual scenes. In contrast, spatially oriented textual hierarchies, where each part describes a distinct image region, align naturally with visual hierarchies.
• Figure 3: CAFT demonstrates fine-grained, grounded vision-language understanding.a) Among visually similar bus images, the baseline method, FLAIR, wavers among look-alikes, whereas our proposed CAFT selects the correct match with a clearly higher retrieval score. b) This improvement arises from representations that attend individual sub-captions to their corresponding image regions, emerging naturally without explicit supervision.
• Figure 4: Overview of CAFT. The model constructs hierarchical representations for both vision and language, aligning them at matched granularities. Vision Branch: Starting from superpixel tokens, the model performs fine-to-coarse scene parsing via progressive token grouping interleaved with ViT blocks. Language Branch: A Sub-caption Transformer encodes each text chunk independently, followed by a Whole-caption Transformer that aggregates them into a holistic embedding. Alignment: We establish a bottom-up hierarchy where mid-level visual features align with sub-caption embeddings for localized grounding (aligning trees), while top-level visual features align with whole-caption embeddings to capture global scene semantics (aligning forest).
• Figure 5: CAFT exhibits strong scaling behavior. We compare CAFT (Ours) with FLAIR (Baseline) across increasing training set sizes (3M, 12M, 15M, and 30M). As the training data scales, our CAFT consistently outperforms the baseline, indicating robust scaling behavior enabled by our hierarchical inductive bias.