Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes

Koen Kraaijveld, Yifan Jiang, Kaixin Ma, Filip Ilievski

TL;DR

COLUMBUS introduces a synthetic, multimodal benchmark for visual lateral thinking by framing rebus puzzles as MCQ VQA tasks generated through a taxonomy-driven pipeline. It defines 18 latent rules, renders puzzles as directed attributed graphs, and samples distractors via hybrid orthographic-semantic similarity, yielding 1,005 puzzles with text and icon variants. Experimental evaluation across diverse vision-language models shows a substantial gap to human ability, with performance significantly boosted when models receive explicit graph-level descriptions but still far from human accuracy. The study highlights the need for global puzzle understanding and abstraction, and points to avenues for expanding scale, balancing puzzle types, and extending the methodology to broader multimodal formats. Overall, COLUMBUS provides a rigorous framework to assess and improve multimodal lateral reasoning in AI systems.

Abstract

While visual question-answering (VQA) benchmarks have catalyzed the development of reasoning techniques, they have focused on vertical thinking. Effective problem-solving also necessitates lateral thinking, which remains understudied in AI and has not been used to test visual perception systems. To bridge this gap, we formulate visual lateral thinking as a multiple-choice question-answering task and describe a three-step taxonomy-driven methodology for instantiating task examples. Then, we develop COLUMBUS, a synthetic benchmark that applies the task pipeline to create QA sets with text and icon rebus puzzles based on publicly available collections of compounds and common phrases. COLUMBUS comprises over 1,000 puzzles, each with four answer candidates. While the SotA vision-language models (VLMs) achieve decent performance, our evaluation demonstrates a substantial gap between humans and models. VLMs benefit from human-curated descriptions but struggle to self-generate such representations at the right level of abstraction.

COLUMBUS: Evaluating COgnitive Lateral Understanding through Multiple-choice reBUSes

TL;DR

COLUMBUS introduces a synthetic, multimodal benchmark for visual lateral thinking by framing rebus puzzles as MCQ VQA tasks generated through a taxonomy-driven pipeline. It defines 18 latent rules, renders puzzles as directed attributed graphs, and samples distractors via hybrid orthographic-semantic similarity, yielding 1,005 puzzles with text and icon variants. Experimental evaluation across diverse vision-language models shows a substantial gap to human ability, with performance significantly boosted when models receive explicit graph-level descriptions but still far from human accuracy. The study highlights the need for global puzzle understanding and abstraction, and points to avenues for expanding scale, balancing puzzle types, and extending the methodology to broader multimodal formats. Overall, COLUMBUS provides a rigorous framework to assess and improve multimodal lateral reasoning in AI systems.

Abstract

While visual question-answering (VQA) benchmarks have catalyzed the development of reasoning techniques, they have focused on vertical thinking. Effective problem-solving also necessitates lateral thinking, which remains understudied in AI and has not been used to test visual perception systems. To bridge this gap, we formulate visual lateral thinking as a multiple-choice question-answering task and describe a three-step taxonomy-driven methodology for instantiating task examples. Then, we develop COLUMBUS, a synthetic benchmark that applies the task pipeline to create QA sets with text and icon rebus puzzles based on publicly available collections of compounds and common phrases. COLUMBUS comprises over 1,000 puzzles, each with four answer candidates. While the SotA vision-language models (VLMs) achieve decent performance, our evaluation demonstrates a substantial gap between humans and models. VLMs benefit from human-curated descriptions but struggle to self-generate such representations at the right level of abstraction.
Paper Structure (30 sections, 10 figures, 10 tables)

This paper contains 30 sections, 10 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology for Visual Lateral Tasks
  4. Taxonomy of Latent Rules
  5. Puzzle Rendering
  6. Distractor Sampling
  7. The COLUMBUS Benchmark
  8. Experimental Design
  9. Model Families.
  10. Results
  11. Overall Performance
  12. Impact of Structural Reasoning
  13. Model Sensitivity to Input Information
  14. Rule-based Analysis
  15. VLM Generation of Puzzles
  16. ...and 15 more sections

Figures (10)

  • Figure 1: Left: vertical thinking puzzle from Machine Number Sense zhang_2020. Right: lateral thinking rebus puzzle from our COLUMBUS benchmark.
  • Figure 2: Methodology for visual lateral thinking tasks.
  • Figure 3: Three taxonomies that classify and organize the individual (top), relational (bottom left), and modifier (bottom right) rules used to manipulate the appearance and position of elements in a rebus puzzle. For each rule, we present an example puzzle and its answer, both taken directly from COLUMBUS.
  • Figure 4: Two examples of directed attributed graphs (left) representing rebus puzzles (right).
  • Figure 5: Results for four prompts that supply the model with increasing information for COLUMBUS-text (left) and COLUMBUS-icon (right) (averaged across three runs). The best-performing model from the following types is shown: open-source non-instruction VLM (Fuyu-8b), open-source instruction VLM (BLIP-2 Flan-T5-XXL), closed-source VLM (GPT-4o), and text-only LLM (Mistral-7b), as well as human accuracy.
  • ...and 5 more figures