SPhyR: Spatial-Physical Reasoning Benchmark on Material Distribution

TL;DR

SPhyR introduces a 2D topology-optimization-inspired benchmark to evaluate how well Large Language Models can reason about material distribution under boundary loads without simulation tools. It combines grid-based inputs with ground-truth minimum-compliance layouts derived from TO solvers and assesses models using reconstruction, topological validity, and physics-approximation metrics, including the FPCEff measure for load-path efficiency. The study reveals that state-of-the-art LLMs exhibit strong global reasoning on easy tasks but struggle with continuous optimization, often over-building or smearing material, and show gravity biases under rotated conditions, indicating a gap between linguistic reasoning and grounded physical understanding. The results highlight the need for integrating geometric constraints and physics-informed priors into LLM prompts and architectures to enable reliable, engineering-relevant spatial reasoning in language models. SPhyR thus provides a concrete, scalable platform to drive progress in physically grounded AI that can assist in design and analysis tasks beyond pure language capabilities.

Abstract

We introduce a novel dataset designed to benchmark the physical and spatial reasoning capabilities of Large Language Models (LLM) based on topology optimization, a method for computing optimal material distributions within a design space under prescribed loads and supports. In this dataset, LLMs are provided with conditions such as 2D boundary, applied forces and supports, and must reason about the resulting optimal material distribution. The dataset includes a variety of tasks, ranging from filling in masked regions within partial structures to predicting complete material distributions. Solving these tasks requires understanding the flow of forces and the required material distribution under given constraints, without access to simulation tools or explicit physical models, challenging models to reason about structural stability and spatial organization. Our dataset targets the evaluation of spatial and physical reasoning abilities in 2D settings, offering a complementary perspective to traditional language and logic benchmarks.

Figures (19)

• Figure 1: Topology Optimization is used to calculate material distribution. Masking individual cells, rows, columns or the complete distribution space offer challenging spatial physical reasoning tasks.
• Figure 2: Prompt template used across all tasks and difficulty levels, showing instructions and grid format as served to models for evaluation.
• Figure 3: Overview of task variations: predicting material distributions for N random cells, rows, columns, or full structures for easy (binary) and hard (continous) difficulties.
• Figure 4: Example 2D topology optimization samples from the SPhyR dataset.
• Figure 5: Example 3D topology optimization samples included for future benchmark extensions.