TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoning

TL;DR

TangramPuzzle introduces a geometry-grounded benchmark for evaluating compositional spatial reasoning in Multimodal LLMs, replacing ambiguous visual matching with exact geometric constraints via the Tangram Construction Expression (TCE). It provides two tasks—Outline Prediction and End-to-End Tangram Solution Generation—under a rigorous constraint-based evaluation framework that uses exact coordinates, non-overlap, and connectivity checks, plus IoU and Hausdorff distance for shape fidelity. The paper details a multi-stage data pipeline, including symbolic normalization and human-in-the-loop validation, and reports that state-of-the-art models struggle to satisfy geometric constraints even when silhouette quality is high, with humans achieving near-perfect solutions. These findings highlight a gap between perceptual alignment and rigorous geometric reasoning, offering a standardized, reusable platform to drive progress in precise spatial understanding for MLLMs. The work has practical impact by enabling robust evaluation of spatial cognition in AI systems and guiding future research toward models that truly integrate symbolic geometry with visual grounding, while also outlining ethical considerations and extension paths to 3D geometry.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual recognition and semantic understanding. Nevertheless, their ability to perform precise compositional spatial reasoning remains largely unexplored. Existing benchmarks often involve relatively simple tasks and rely on semantic approximations or coarse relative positioning, while their evaluation metrics are typically limited and lack rigorous mathematical formulations. To bridge this gap, we introduce TangramPuzzle, a geometry-grounded benchmark designed to evaluate compositional spatial reasoning through the lens of the classic Tangram game. We propose the Tangram Construction Expression (TCE), a symbolic geometric framework that grounds tangram assemblies in exact, machine-verifiable coordinate specifications, to mitigate the ambiguity of visual approximation. We design two complementary tasks: Outline Prediction, which demands inferring global shapes from local components, and End-to-End Code Generation, which requires solving inverse geometric assembly problems. We conduct extensive evaluation experiments on advanced open-source and proprietary models, revealing an interesting insight: MLLMs tend to prioritize matching the target silhouette while neglecting geometric constraints, leading to distortions or deformations of the pieces.

Figures (5)

• Figure 1: Illustration of the two spatial reasoning tasks. (a) Task 1 global shape inference; (b) Task 2 constrained shape decomposition.
• Figure 2: Compact representation of a TCE instance.
• Figure 3: Overview of the TangramPuzzle data construction pipeline.
• Figure 4: Visualization of a case from Task 2.
• Figure 5: Annotation interface for TangramPuzzle.