Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraints

Rui Xu, Dakuan Lu, Zicheng Zhao, Xiaoyu Tan, Xintao Wang, Siyu Yuan, Jiangjie Chen, Yinghui Xu

TL;DR

OrigamiSpace introduces a constraint-rich benchmark and a 350-entry origami dataset to evaluate multimodal LLMs on multi-step spatial reasoning. It features four tasks—Pattern Prediction, Multi-step Spatial Reasoning, Spatial Relationship Prediction, and End-to-End CP Code Generation—alongside an enhanced origami compiler and an interactive environment for CP code development, including reinforcement learning investigations. Across open- and closed-source models, results show that even strong models struggle with multi-step reasoning and strict geometric constraints, though iterative interaction and RL-based training substantially improve performance. The work demonstrates the potential of origami as a rigorous testbed for spatial intelligence in MLLMs and highlights avenues for improving constraint adherence via feedback-driven learning.

Abstract

Spatial reasoning is a key capability in the field of artificial intelligence, especially crucial in areas such as robotics, computer vision, and natural language understanding. However, evaluating the ability of multimodal large language models(MLLMs) in complex spatial reasoning still faces challenges, particularly in scenarios requiring multi-step reasoning and precise mathematical constraints. This paper introduces ORIGAMISPACE, a new dataset and benchmark designed to evaluate the multi-step spatial reasoning ability and the capacity to handle mathematical constraints of MLLMs through origami tasks. The dataset contains 350 data instances,each comprising a strictly formatted crease pattern (CP diagram), the Compiled Flat Pattern, the complete Folding Process, and the final Folded Shape Image. We propose four evaluation tasks: Pattern Prediction, Multi-step Spatial Reasoning, Spatial Relationship Prediction, and End-to-End CP Code Generation. For the CP code generation task, we design an interactive environment and explore the possibility of using reinforcement learning methods to train MLLMs. Through experiments on existing MLLMs, we initially reveal the strengths and weaknesses of these models in handling complex spatial reasoning tasks.

ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraints

TL;DR

OrigamiSpace introduces a constraint-rich benchmark and a 350-entry origami dataset to evaluate multimodal LLMs on multi-step spatial reasoning. It features four tasks—Pattern Prediction, Multi-step Spatial Reasoning, Spatial Relationship Prediction, and End-to-End CP Code Generation—alongside an enhanced origami compiler and an interactive environment for CP code development, including reinforcement learning investigations. Across open- and closed-source models, results show that even strong models struggle with multi-step reasoning and strict geometric constraints, though iterative interaction and RL-based training substantially improve performance. The work demonstrates the potential of origami as a rigorous testbed for spatial intelligence in MLLMs and highlights avenues for improving constraint adherence via feedback-driven learning.

Abstract

Spatial reasoning is a key capability in the field of artificial intelligence, especially crucial in areas such as robotics, computer vision, and natural language understanding. However, evaluating the ability of multimodal large language models(MLLMs) in complex spatial reasoning still faces challenges, particularly in scenarios requiring multi-step reasoning and precise mathematical constraints. This paper introduces ORIGAMISPACE, a new dataset and benchmark designed to evaluate the multi-step spatial reasoning ability and the capacity to handle mathematical constraints of MLLMs through origami tasks. The dataset contains 350 data instances,each comprising a strictly formatted crease pattern (CP diagram), the Compiled Flat Pattern, the complete Folding Process, and the final Folded Shape Image. We propose four evaluation tasks: Pattern Prediction, Multi-step Spatial Reasoning, Spatial Relationship Prediction, and End-to-End CP Code Generation. For the CP code generation task, we design an interactive environment and explore the possibility of using reinforcement learning methods to train MLLMs. Through experiments on existing MLLMs, we initially reveal the strengths and weaknesses of these models in handling complex spatial reasoning tasks.

Paper Structure

This paper contains 63 sections, 6 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Spatial Reasoning Benchmarks
  4. Computational Origami
  5. OrigamiSpace Dataset
  6. Data Collection
  7. Compiler
  8. Dataset Statistics
  9. Task
  10. Pattern Prediction
  11. Multi-step Spatial Reasoning
  12. Spatial Relationship Prediction
  13. End-to-End CP Code Generation
  14. Experiments
  15. Models
  16. ...and 48 more sections

Figures (3)

  • Figure 1: An example data instance from OrigamiSpace includes: CP Diagram, Compiled Flat Pattern, Folded Shape Image, and Folding Process, where the CP Diagram can be represented in the form of CP Code.
  • Figure 2: Data examples of the four tasks. The first three tasks are in a multiple-choice format, and the fourth task is a code generation task.
  • Figure 3: The impact of interaction rounds on the compilation pass rate and total score of different models.