Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models

Nicoleta-Nina Basoc, Adrian Cosma, Emilian Radoi

TL;DR

It is shown that a simple human-inspired preprocessing step recovers a meaningful portion of performance for several distortion types, motivating perception-aware preprocessing and tool-use as practical strategies for improving VLM robustness.

Abstract

Vision-language models (VLMs) achieve strong benchmark results, yet can exhibit systematic perceptual weaknesses: structured, large changes to pixel values can cause confident yet nonsensical predictions, even when the underlying scene remains easily recognizable to humans. We study this gap using Spatial Colour Mixing, a programmatic family of colour distortions that overlays structured patterns (in both RGB and Ostwald colour systems) onto natural images. We introduce a framework of eight spatial colour mixing variants and evaluate nine VLMs across three model families on four datasets. Across models and datasets, accuracy degrades sharply with increasing distortion, and scaling the language model does not reliably mitigate the failure. In a human study with 61 participants on an animal recognition dataset, humans substantially outperform VLMs under the same distortions. Finally, we show that a simple human-inspired preprocessing step recovers a meaningful portion of performance for several distortion types, motivating perception-aware preprocessing and tool-use as practical strategies for improving VLM robustness.

Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models

TL;DR

It is shown that a simple human-inspired preprocessing step recovers a meaningful portion of performance for several distortion types, motivating perception-aware preprocessing and tool-use as practical strategies for improving VLM robustness.

Abstract

Vision-language models (VLMs) achieve strong benchmark results, yet can exhibit systematic perceptual weaknesses: structured, large changes to pixel values can cause confident yet nonsensical predictions, even when the underlying scene remains easily recognizable to humans. We study this gap using Spatial Colour Mixing, a programmatic family of colour distortions that overlays structured patterns (in both RGB and Ostwald colour systems) onto natural images. We introduce a framework of eight spatial colour mixing variants and evaluate nine VLMs across three model families on four datasets. Across models and datasets, accuracy degrades sharply with increasing distortion, and scaling the language model does not reliably mitigate the failure. In a human study with 61 participants on an animal recognition dataset, humans substantially outperform VLMs under the same distortions. Finally, we show that a simple human-inspired preprocessing step recovers a meaningful portion of performance for several distortion types, motivating perception-aware preprocessing and tool-use as practical strategies for improving VLM robustness.
Paper Structure (17 sections, 8 figures, 1 table)

This paper contains 17 sections, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-language models
  4. Evaluating VLM visual perception through illusions
  5. Human visual perception
  6. Method
  7. Spatial colour mixing illusions
  8. Datasets
  9. Evaluation protocol
  10. Human evaluation
  11. Preprocessing via down/up rescaling and blurring
  12. Experiments and Results
  13. Main results
  14. Human evaluation
  15. Human-inspired low-pass preprocessing
  16. ...and 2 more sections

Figures (8)

  • Figure 1: What animals are shown in these examples? Readers are invited to distance themselves from the screen / zoom in and observe how their perception changes. Best viewed on a colour display.
  • Figure 2: Examples of each of our 8 spatial colour mixing illusions. Best viewed on a colour display.
  • Figure 3: Examples of the effect of the distortion degree on four of our proposed colour mixing illusions. Best viewed on a coloured screen.
  • Figure 4: Main results. Accuracy as a function of distortion degree for 9 VLMs across four datasets (Animals, Artworks, Landmarks, MME) under eight Spatial Colour Mixing variants. Columns group models by family (Qwen3-VL, LLaVA, Gemma3); within each panel, line style indicates model scale and colour indicates the illusion type. Across datasets, accuracy degrades sharply even at low distortion degrees, and differences are driven more by model family than by language-model scale.
  • Figure 5: Mean per-image accuracy on the Animals dataset for three distortion degrees (2, 5, 12) and three representative illusion types (SCMix-1, Ostwald Checker, Ostwald Random). Bars compare human responses to the aggregated performance of the nine evaluated VLMs.
  • ...and 3 more figures