Representations of Text and Images Align From Layer One

TL;DR

The paper shows that text and image representations in adapter-based vision-language models align from the very first layer for many concepts, challenging the prevailing view of mid-to-late alignment. It introduces a dataset-free synthesis method, Direct Ascents Synthesis (DAS), to generate prototype images whose layer-$\ell$ representations align with textual concept directions, validated by transfer to independent models. Text concepts are represented via steering vectors, while image representations use a weighted patch aggregation; the optimisation targets transfer and uses multi-resolution perturbations with augmentations. Across 100+ concepts and seven layers in Gemma 3, many categories yield recognisable prototypes at layer 1, though a middle-layer collapse is observed in Gemma that is not universal across models. The work contributes a new interpretability tool and provides evidence that cross-modal grounding emerges early, with implications for understanding and visualising how vision-language models ground abstract concepts in the visual world.

Abstract

We show that for a variety of concepts in adapter-based vision-language models, the representations of their images and their text descriptions are meaningfully aligned from the very first layer. This contradicts the established view that such image-text alignment only appears in late layers. We show this using a new synthesis-based method inspired by DeepDream: given a textual concept such as "Jupiter", we extract its concept vector at a given layer, and then use optimisation to synthesise an image whose representation aligns with that vector. We apply our approach to hundreds of concepts across seven layers in Gemma 3, and find that the synthesised images often depict salient visual features of the targeted textual concepts: for example, already at layer 1, more than 50 % of images depict recognisable features of animals, activities, or seasons. Our method thus provides direct, constructive evidence of image-text alignment on a concept-by-concept and layer-by-layer basis. Unlike previous methods for measuring multimodal alignment, our approach is simple, fast, and does not require auxiliary models or datasets. It also offers a new path towards model interpretability, by providing a way to visualise a model's representation space by backtracing through its image processing components.

Figures (9)

• Figure 1: Top: Given a text description of a concept (e.g., "Jupiter") and a target layer in a vision-language model, our method synthesises an image whose representation aligns with that of the target word in that layer. This enables studying how the models bridge between the modalities. Bottom: Surprisingly, as early as layer 1, the synthesised images are often recognisable as depicting the target concepts, demonstrating that representations of text and images align much earlier than previously thought.
• Figure 2: Our image synthesis pipeline based on direct ascent synthesis fort2025directascentsynthesisrevealing. The final image consists of a base image (neutral grey) plus a trainable perturbation. (1) Instead of training the perturbation pixels directly, we re-parametrise the perturbation via a set of trainable multi-resolution components. (2--3) All multi-resolution components are upscaled to the target size, summed, and added to the base image. (4) The resulting image is augmented with random shifts and noise.
• Figure 3: The proportion of synthesised images that GPT-5 recognised as depicting their target concept, per category and per layer. Results shown are for evaluation with category hints; without hints, only animal concepts achieve reliable recognition. Example images show best results per category. Shaded regions: 95% CI. Images cropped for visibility.
• Figure 4: Images generated on InternVL 3 8B maintain high recognisability even in middle layers, unlike ones generated on Gemma 3 4B where recognisability drops to near zero in layers 15--20 (\ref{['fig:dictionary']}). Evaluation with category hints; example images show best results.
• Figure 5: Some synthesised images do not visually depict the target concept but instead contain embedded text that describes the concept, suggesting the model can "read images" from layer 1.