Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?

Michael Aerni, Joshua Swanson, Kristina Nikolić, Florian Tramèr

TL;DR

Modal aphasia reveals a robust cross-modal dissociation in unified multimodal models: they can visually reproduce learned concepts with high fidelity but struggle to articulate the same concepts in text. Through frontier-model demonstrations and open-weight controlled experiments, the work shows a consistent image-text gap with text descriptions exhibiting substantially more hallucinations and lower accuracy. The findings persist across synthetic-face and abstract-concept datasets, suggesting a fundamental limitation in how current architectures store and transfer cross-modal knowledge. A safety case study further indicates that text-only safeguards may fail to block unsafe concepts that remain accessible via other modalities, underscoring the need for models that integrate visualization into reasoning or adopt cross-modal alignment strategies. Overall, the work highlights a core challenge in achieving truly unified multimodal understanding and provides reproducible experimental resources to probe cross-modal knowledge transfer.

Abstract

We present modal aphasia, a systematic dissociation in which current unified multimodal models accurately memorize concepts visually but fail to articulate them in writing, despite being trained on images and text simultaneously. For one, we show that leading frontier models can generate near-perfect reproductions of iconic movie artwork, but confuse crucial details when asked for textual descriptions. We corroborate those findings through controlled experiments on synthetic datasets in multiple architectures. Our experiments confirm that modal aphasia reliably emerges as a fundamental property of current unified multimodal models, not just as a training artifact. In practice, modal aphasia can introduce vulnerabilities in AI safety frameworks, as safeguards applied to one modality may leave harmful concepts accessible in other modalities. We demonstrate this risk by showing how a model aligned solely on text remains capable of generating unsafe images.

Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?

TL;DR

Modal aphasia reveals a robust cross-modal dissociation in unified multimodal models: they can visually reproduce learned concepts with high fidelity but struggle to articulate the same concepts in text. Through frontier-model demonstrations and open-weight controlled experiments, the work shows a consistent image-text gap with text descriptions exhibiting substantially more hallucinations and lower accuracy. The findings persist across synthetic-face and abstract-concept datasets, suggesting a fundamental limitation in how current architectures store and transfer cross-modal knowledge. A safety case study further indicates that text-only safeguards may fail to block unsafe concepts that remain accessible via other modalities, underscoring the need for models that integrate visualization into reasoning or adopt cross-modal alignment strategies. Overall, the work highlights a core challenge in achieving truly unified multimodal understanding and provides reproducible experimental resources to probe cross-modal knowledge transfer.

Abstract

We present modal aphasia, a systematic dissociation in which current unified multimodal models accurately memorize concepts visually but fail to articulate them in writing, despite being trained on images and text simultaneously. For one, we show that leading frontier models can generate near-perfect reproductions of iconic movie artwork, but confuse crucial details when asked for textual descriptions. We corroborate those findings through controlled experiments on synthetic datasets in multiple architectures. Our experiments confirm that modal aphasia reliably emerges as a fundamental property of current unified multimodal models, not just as a training artifact. In practice, modal aphasia can introduce vulnerabilities in AI safety frameworks, as safeguards applied to one modality may leave harmful concepts accessible in other modalities. We demonstrate this risk by showing how a model aligned solely on text remains capable of generating unsafe images.
Paper Structure (54 sections, 12 figures, 6 tables)

This paper contains 54 sections, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Modal Aphasia in Frontier Models
  4. Setup
  5. Evaluation
  6. Error types.
  7. Results
  8. Image generation is more accurate than description.
  9. High hallucination rates in textual descriptions.
  10. Image generation produces no major hallucinations
  11. Controlled Experiments on Open-Weight Models
  12. Setup
  13. Unified open-weight models
  14. Training
  15. Evaluation
  16. ...and 39 more sections

Figures (12)

  • Figure 1: ChatGPT-5 generates accurate movie posters but fails to describe them. We ask ChatGPT-5 to generate a set of popular movie posters and independently query it to describe the same posters in text. While the model excels at reproducing the artworks visually, it consistently fails to describe them verbally. We term this phenomenon modal aphasia. All cases use the web interface without search and provide only the movie titles.
  • Figure 2: ChatGPT-5 suffers from modal aphasia. We use ChatGPT-5 to generate famous movie posters from memory, independently as images and as textual descriptions. (a) On average, textual descriptions of posters contain over $7\times$ more errors than the corresponding generated images, with a majority of errors being hallucinations (fabricated or wrong details). (b) We detect major hallucinations (e.g., fabricated characters) exclusively in text descriptions, which contain around 95% of all anticipated major hallucinations on average. In contrast, generated images only contain minor hallucinations (e.g., incorrect details)---and $5\times$ fewer than the corresponding textual descriptions. Error bars show the standard error across three evaluation runs.
  • Figure 3: Example images and prompts. Generated faces with their randomly assigned names (top); abstract synthetic concepts with the fake names for each concept (bottom).
  • Figure 4: Models can generate accurate faces but not describe their features. We train models to generate synthetic portraits of fictional people. Given only a person's name, we then measure how accurately models generate and describe that person's eye color, hair color, hairstyle, and accessories. (a) Both fine-tuned model types manage to generate accurate portraits, but perform random guessing (between 20% and 25% accuracy) when trying to describe the same faces. Bars report the mean over three training seeds, lines the standard error. (b) There is no clear correlation between a model's accuracy when generating face images vs. describing them in text. See \ref{['app:aux_controlled']} for additional results.
  • Figure 5: Models generalize to abstract concepts visually but not verbally. We fine-tune models to generate a combination of abstract visual concepts given their (made-up) names. (a) Both model types achieve high accuracy on seen (Train) and unseen (Test) combination of concepts when generating images, but underperform when describing the same concepts verbally. (b) We observe different degrees of modal aphasia for different types of concepts. For shapes, Janus-Pro outperforms the random guessing baseline of around 14%, but performs worse than random on positions (25% baseline). We only report individual accuracies on Janus for brevity; see \ref{['app:aux_controlled']} for Harmon and the faces dataset. Bars show the mean over three seeds, lines the standard error.
  • ...and 7 more figures