Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions

TL;DR

This work introduces Illusory VQA, a task and four synthetic datasets (IllusionMNIST, IllusionFashionMNIST, IllusionAnimals, IllusionChar) to probe multimodal models on visual illusions. It demonstrates that illusory content degrades zero-shot performance across state-of-the-art models, but a fixed Gaussian blur low-pass preprocessing can substantially improve robustness, with BLIP-2 on IllusionAnimals even surpassing human performance after filtering. The authors also report gains from fine-tuning on illusory data and discuss the potential for adaptive learnable filters and expanded illusory datasets. Overall, the study highlights gaps between human and model perception of illusions and suggests practical preprocessing and future architectural directions to bridge this gap.

Abstract

In recent years, Visual Question Answering (VQA) has made significant strides, particularly with the advent of multimodal models that integrate vision and language understanding. However, existing VQA datasets often overlook the complexities introduced by image illusions, which pose unique challenges for both human perception and model interpretation. In this study, we introduce a novel task called Illusory VQA, along with four specialized datasets: IllusionMNIST, IllusionFashionMNIST, IllusionAnimals, and IllusionChar. These datasets are designed to evaluate the performance of state-of-the-art multimodal models in recognizing and interpreting visual illusions. We assess the zero-shot performance of various models, fine-tune selected models on our datasets, and propose a simple yet effective solution for illusion detection using Gaussian and blur low-pass filters. We show that this method increases the performance of models significantly and in the case of BLIP-2 on IllusionAnimals without any fine-tuning, it outperforms humans. Our findings highlight the disparity between human and model perception of illusions and demonstrate that fine-tuning and specific preprocessing techniques can significantly enhance model robustness. This work contributes to the development of more human-like visual understanding in multimodal models and suggests future directions for adapting filters using learnable parameters.

Figures (39)

• Figure 1: Illustration of illusion in image
• Figure 2: Examples of answers of GPT-4o and Gemini to raw images, illusory images, and applied filter one. Through the process of image minification, the human eye becomes adept at uncovering the hidden illusions within. (a): An example of IllusionMNIST, (b): An example of IllusionFashionMNIST, (c): An example of IllusionAnimals, (d): An example of IllusionChar, (e): An example of 'No illusion' class
• Figure 3: Demonstraion of our pipeline for generating and evaluating datasets
• Figure 4: An example of task definition
• Figure 5: An example of task definition