Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SMSP: A Plug-and-Play Strategy of Multi-Scale Perception for MLLMs to Perceive Visual Illusions

Jinzhe Tu, Ruilei Guo, Zihan Guo, Junxiao Yang, Shiyao Cui, Minlie Huang

Abstract

Recent works have shown that Multimodal Large Language Models (MLLMs) are highly vulnerable to hidden-pattern visual illusions, where the hidden content is imperceptible to models but obvious to humans. This deficiency highlights a perceptual misalignment between current MLLMs and humans, and also introduces potential safety concerns. To systematically investigate this failure, we introduce IlluChar, a comprehensive and challenging illusion dataset, and uncover a key underlying mechanism for the models' failure: high-frequency attention bias, where the models are easily distracted by high-frequency background textures in illusion images, causing them to overlook hidden patterns. To address the issue, we propose the Strategy of Multi-Scale Perception (SMSP), a plug-and-play framework that aligns with human visual perceptual strategies. By suppressing distracting high-frequency backgrounds, SMSP generates images closer to human perception. Our experiments demonstrate that SMSP significantly improves the performance of all evaluated MLLMs on illusion images, for instance, increasing the accuracy of Qwen3-VL-8B-Instruct from 13.0% to 84.0%. Our work provides novel insights into MLLMs' visual perception, and offers a practical and robust solution to enhance it. Our code is publicly available at https://github.com/Tujz2023/SMSP.

SMSP: A Plug-and-Play Strategy of Multi-Scale Perception for MLLMs to Perceive Visual Illusions

Abstract

Recent works have shown that Multimodal Large Language Models (MLLMs) are highly vulnerable to hidden-pattern visual illusions, where the hidden content is imperceptible to models but obvious to humans. This deficiency highlights a perceptual misalignment between current MLLMs and humans, and also introduces potential safety concerns. To systematically investigate this failure, we introduce IlluChar, a comprehensive and challenging illusion dataset, and uncover a key underlying mechanism for the models' failure: high-frequency attention bias, where the models are easily distracted by high-frequency background textures in illusion images, causing them to overlook hidden patterns. To address the issue, we propose the Strategy of Multi-Scale Perception (SMSP), a plug-and-play framework that aligns with human visual perceptual strategies. By suppressing distracting high-frequency backgrounds, SMSP generates images closer to human perception. Our experiments demonstrate that SMSP significantly improves the performance of all evaluated MLLMs on illusion images, for instance, increasing the accuracy of Qwen3-VL-8B-Instruct from 13.0% to 84.0%. Our work provides novel insights into MLLMs' visual perception, and offers a practical and robust solution to enhance it. Our code is publicly available at https://github.com/Tujz2023/SMSP.
Paper Structure (41 sections, 6 equations, 12 figures, 11 tables, 1 algorithm)

This paper contains 41 sections, 6 equations, 12 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Visual Illusions
  4. Analysis and Mitigation of Hidden-Pattern Visual Illusions
  5. IlluChar: A Character-Based Illusion Dataset
  6. Dataset Overview
  7. Dataset Construction
  8. Evaluation and Results
  9. Mechanism: High-Frequency Attention Bias
  10. Mitigation Method
  11. Inspiration: Human Perceptual Strategies
  12. The Strategy of Multi-Scale Perception
  13. Perception Module
  14. Multi-Scale Strategy
  15. Experiments
  16. ...and 26 more sections

Figures (12)

  • Figure 1: Top: An illusion image with an emergency signal. The model's attention is dispersed by the background and fails to detect it, while humans can identify it by adjusting their perception. Bottom: After processing the image to simulate such perceptual adjustments, the model can focus on the signal and successfully recognize it.
  • Figure 2: Examples across different categories in IlluChar.
  • Figure 3: Spectral energy distribution comparison between original and illusion images.
  • Figure 4: Analysis of the variance in model's attention distribution. Left: A quantitative analysis of the model’s high-attention (regions with top $20\%$ attention scores) distribution. Right: Three representative examples.
  • Figure 5: An outline of the Strategy of Multi-scale Perception (SMSP). Two examples are provided to demonstrate the whole process and illustrate how models identify hidden characters with the help of SMSP.
  • ...and 7 more figures