Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

REBUS: A Robust Evaluation Benchmark of Understanding Symbols

Andrew Gritsevskiy, Arjun Panickssery, Aaron Kirtland, Derik Kauffman, Hans Gundlach, Irina Gritsevskaya, Joe Cavanagh, Jonathan Chiang, Lydia La Roux, Michelle Hung

TL;DR

REBUS introduces a robust multimodal benchmark for evaluating symbol understanding through image-based wordplay. The dataset comprises 333 hand-crafted rebuses across 13 categories, demanding image grounding, spelling manipulation, hypothesis testing, and world knowledge, thereby exposing gaps in current multimodal LLMs. Empirical results show proprietary models (notably GPT-4o) achieving the best overall performance (~42%), but accuracy collapses on harder puzzles, and open-source methods lag substantially, highlighting persistent reasoning and faithfulness challenges. By analyzing calibration, faithfulness, and human baselines, REBUS provides a comprehensive framework to diagnose core weaknesses in multimodal reasoning and guides future improvements in grounding, explainability, and knowledge integration.

Abstract

We propose a new benchmark evaluating the performance of multimodal large language models on rebus puzzles. The dataset covers 333 original examples of image-based wordplay, cluing 13 categories such as movies, composers, major cities, and food. To achieve good performance on the benchmark of identifying the clued word or phrase, models must combine image recognition and string manipulation with hypothesis testing, multi-step reasoning, and an understanding of human cognition, making for a complex, multimodal evaluation of capabilities. We find that GPT-4o significantly outperforms all other models, followed by proprietary models outperforming all other evaluated models. However, even the best model has a final accuracy of only 42\%, which goes down to just 7\% on hard puzzles, highlighting the need for substantial improvements in reasoning. Further, models rarely understand all parts of a puzzle, and are almost always incapable of retroactively explaining the correct answer. Our benchmark can therefore be used to identify major shortcomings in the knowledge and reasoning of multimodal large language models.

REBUS: A Robust Evaluation Benchmark of Understanding Symbols

TL;DR

REBUS introduces a robust multimodal benchmark for evaluating symbol understanding through image-based wordplay. The dataset comprises 333 hand-crafted rebuses across 13 categories, demanding image grounding, spelling manipulation, hypothesis testing, and world knowledge, thereby exposing gaps in current multimodal LLMs. Empirical results show proprietary models (notably GPT-4o) achieving the best overall performance (~42%), but accuracy collapses on harder puzzles, and open-source methods lag substantially, highlighting persistent reasoning and faithfulness challenges. By analyzing calibration, faithfulness, and human baselines, REBUS provides a comprehensive framework to diagnose core weaknesses in multimodal reasoning and guides future improvements in grounding, explainability, and knowledge integration.

Abstract

We propose a new benchmark evaluating the performance of multimodal large language models on rebus puzzles. The dataset covers 333 original examples of image-based wordplay, cluing 13 categories such as movies, composers, major cities, and food. To achieve good performance on the benchmark of identifying the clued word or phrase, models must combine image recognition and string manipulation with hypothesis testing, multi-step reasoning, and an understanding of human cognition, making for a complex, multimodal evaluation of capabilities. We find that GPT-4o significantly outperforms all other models, followed by proprietary models outperforming all other evaluated models. However, even the best model has a final accuracy of only 42\%, which goes down to just 7\% on hard puzzles, highlighting the need for substantial improvements in reasoning. Further, models rarely understand all parts of a puzzle, and are almost always incapable of retroactively explaining the correct answer. Our benchmark can therefore be used to identify major shortcomings in the knowledge and reasoning of multimodal large language models.
Paper Structure (16 sections, 6 figures, 4 tables)

This paper contains 16 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The REBUS Benchmark
  4. Dataset description
  5. Limitations
  6. Experiments
  7. Baselines
  8. Calibration and in-context answers
  9. Faithfulness
  10. Human baselines
  11. Future Work
  12. Additional examples of faithlessness
  13. Examples of various failure modes
  14. Correct answer, incorrect reasoning
  15. Incorrect reasoning, correct recognition
  16. ...and 1 more sections

Figures (6)

  • Figure 1: Breakdown of rebus characteristics.
  • Figure 2: Our evaluation prompt template.
  • Figure 3: Model performance on the REBUS benchmark.
  • Figure 4: Performance of models on the benchmark over time. Position on the x-axis corresponds to model release date.
  • Figure 5: Calibration curves of GPT-4o, GPT-4V, Gemini Pro 1.5, and Gemini Pro 1.0.
  • ...and 1 more figures