Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChromouVQA: Benchmarking Vision-Language Models under Chromatic Camouflaged Images

Yunfei Zhang, Yizhuo He, Yuanxun Shao, Zhengtao Yao, Haoyan Xu, Junhao Dong, Zhen Yao, Zhikang Dong

TL;DR

ChromouVQA presents a large-scale Ishihara-style camouflaged VQA benchmark to probe vision-language models' figure–ground segregation. It provides a controllable data-generation pipeline producing 70,200 camouflaged images across 61 configurations and nine QA tasks, paired with a model-agnostic contrastive fine-tuning framework that aligns silhouettes with camouflage to recover global shapes. Empirical results show a substantial gap between human and model performance, with contrastive adaptation yielding meaningful gains across backbones, establishing strong baselines for future research. The work offers a compact, reproducible benchmark for evaluating perceptual and reasoning capabilities under chromatic camouflage and outlines clear paths for expansion.

Abstract

Vision-Language Models (VLMs) have advanced multimodal understanding, yet still struggle when targets are embedded in cluttered backgrounds requiring figure-ground segregation. To address this, we introduce ChromouVQA, a large-scale, multi-task benchmark based on Ishihara-style chromatic camouflaged images. We extend classic dot plates with multiple fill geometries and vary chromatic separation, density, size, occlusion, and rotation, recording full metadata for reproducibility. The benchmark covers nine vision-question-answering tasks, including recognition, counting, comparison, and spatial reasoning. Evaluations of humans and VLMs reveal large gaps, especially under subtle chromatic contrast or disruptive geometric fills. We also propose a model-agnostic contrastive recipe aligning silhouettes with their camouflaged renderings, improving recovery of global shapes. ChromouVQA provides a compact, controlled benchmark for reproducible evaluation and extension. Code and dataset are available at https://github.com/Chromou-VQA-Benchmark/Chromou-VQA.

ChromouVQA: Benchmarking Vision-Language Models under Chromatic Camouflaged Images

TL;DR

ChromouVQA presents a large-scale Ishihara-style camouflaged VQA benchmark to probe vision-language models' figure–ground segregation. It provides a controllable data-generation pipeline producing 70,200 camouflaged images across 61 configurations and nine QA tasks, paired with a model-agnostic contrastive fine-tuning framework that aligns silhouettes with camouflage to recover global shapes. Empirical results show a substantial gap between human and model performance, with contrastive adaptation yielding meaningful gains across backbones, establishing strong baselines for future research. The work offers a compact, reproducible benchmark for evaluating perceptual and reasoning capabilities under chromatic camouflage and outlines clear paths for expansion.

Abstract

Vision-Language Models (VLMs) have advanced multimodal understanding, yet still struggle when targets are embedded in cluttered backgrounds requiring figure-ground segregation. To address this, we introduce ChromouVQA, a large-scale, multi-task benchmark based on Ishihara-style chromatic camouflaged images. We extend classic dot plates with multiple fill geometries and vary chromatic separation, density, size, occlusion, and rotation, recording full metadata for reproducibility. The benchmark covers nine vision-question-answering tasks, including recognition, counting, comparison, and spatial reasoning. Evaluations of humans and VLMs reveal large gaps, especially under subtle chromatic contrast or disruptive geometric fills. We also propose a model-agnostic contrastive recipe aligning silhouettes with their camouflaged renderings, improving recovery of global shapes. ChromouVQA provides a compact, controlled benchmark for reproducible evaluation and extension. Code and dataset are available at https://github.com/Chromou-VQA-Benchmark/Chromou-VQA.

Paper Structure

This paper contains 14 sections, 1 equation, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Generation
  4. Unified Task Suite: Visual Reasoning and Recognition
  5. Chromatic Separation Control
  6. Diverse Shapes Filling
  7. Data Statistics
  8. Contrastive Fine-Tuning Framework
  9. Contrastive Fine-Tuning Method
  10. Empirical Results
  11. Evaluation Protocol
  12. Results and Analysis
  13. Ablation Studies
  14. Conclusion

Figures (4)

  • Figure 1: Dataset generation and model inference pipeline (rotation-invariant perception task).
  • Figure 2: Example tasks in ChromouVQA, details are in the Dataset Generation section.
  • Figure 3: Filling algorithm from franciscouzo2025ishihara (left) and our ray casting-based filling method (right).
  • Figure 4: Evaluation Accuracy Under Different Configurations. Each subplot compares model performance across three types of image configurations: (a) CIE color palettes, (b) Ishihara color numbers, (c) Filling shape types.