The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models

TL;DR

This work interrogates nonverbal abstract reasoning in multi-modal LLMs by evaluating 24 models on RPM-style benchmarks (IQ50, RAVEN, CCSE) with semi-isolated setups to separate textual and visual contributions. It reveals a substantial performance gap between open-source and closed-source MLLMs, with GPT-4V showing non-trivial capabilities but still limited by misalignment and surface-level generation biases. The paper identifies textual reasoning and visual awareness as root causes of poor performance and demonstrates that guided prompting, especially corrective hints, can dramatically boost results (up to ~100% in some cases) for closed-source models. It argues for more grounded, scalable evaluation and stronger cross-modal alignment to realize robust nonverbal abstract reasoning in future MLLMs, providing datasets and code to support ongoing research.

Abstract

While large language models (LLMs) are still being adopted to new domains and utilized in novel applications, we are experiencing an influx of the new generation of foundation models, namely multi-modal large language models (MLLMs). These models integrate verbal and visual information, opening new possibilities to demonstrate more complex reasoning abilities at the intersection of the two modalities. However, despite the revolutionizing prospect of MLLMs, our understanding of their reasoning abilities is limited. In this study, we assess the nonverbal abstract reasoning abilities of open-source and closed-source MLLMs using variations of Raven's Progressive Matrices. Our experiments reveal the challenging nature of such problems for MLLMs while showcasing the immense gap between open-source and closed-source models. We also uncover critical shortcomings of visual and textual perceptions, subjecting the models to low-performance ceilings. Finally, to improve MLLMs' performance, we experiment with different methods, such as Chain-of-Thought prompting, leading to a significant (up to 100%) boost in performance. Our code and datasets are available at https://github.com/usc-isi-i2/isi-mmlm-rpm.

Figures (10)

• Figure 1: An example of model's prediction on a sample from the IQ50 dataset. Given a prompt with a visual puzzle (top), the model generates a response that includes its reasoning and the chosen option.
• Figure 2: Zero-shot accuracy concerning the number of parameters using the automatic scoring method. Models are sorted from smallest (left) to largest (right), and those within the same family are colored the same. The red dashed lines indicate the random baselines.
• Figure 3: Zero-shot CoT accuracy on IQ50 using text-only prompts.
• Figure 5: Guided prompting performance of gpt-4v and gemini-pro-vision on IQ50 using different types of hints. Legend: Z-S $\rightarrow$ Zero-shot, Gen $\rightarrow$ General, Sam $\rightarrow$ Sample-specific, and Cor $\rightarrow$ Corrective.
• Figure 6: Zero-shot and symmetrical few-shot accuracy on IQ50. In (a) In-Distribution, the demonstrations are taken from IQ50, while in (b) Out-of-Distribution, the demonstrations are taken from RAVEN-S. Each variation was executed five times with different seeds to mitigate the effect of random sampling. The red dashed lines indicate the random baselines.