LTD-Bench: Evaluating Large Language Models by Letting Them Draw
Liuhao Lin, Ke Li, Zihan Xu, Yuchen Shi, Yulei Qin, Yan Zhang, Xing Sun, Rongrong Ji
TL;DR
LTD-Bench tackles the neglected spatial reasoning dimension of large language models by replacing opaque metrics with directly observable visual outputs. It uses dual paths—generation ( translating text into visual artifacts such as dot matrices or drawing code) and recognition (interpreting visuals to generate language labels)—across Easy, Normal, and Hard levels to probe bidirectional language–spatial mappings. The benchmark demonstrates that state-of-the-art models exhibit substantial gaps in spatial perception and imagination, with deep reasoning boosting recognition more than generation and multimodal models not consistently outperforming text-only baselines. Beyond evaluation, LTD-Bench provides a diagnostic lens into model similarity through stylistic analysis of visual outputs, offering a practical pathway to develop more robust, grounded world models for real-world interaction tasks.
Abstract
Current evaluation paradigms for large language models (LLMs) represent a critical blind spot in AI research--relying on opaque numerical metrics that conceal fundamental limitations in spatial reasoning while providing no intuitive understanding of model capabilities. This deficiency creates a dangerous disconnect between reported performance and practical abilities, particularly for applications requiring physical world understanding. We introduce LTD-Bench, a breakthrough benchmark that transforms LLM evaluation from abstract scores to directly observable visual outputs by requiring models to generate drawings through dot matrices or executable code. This approach makes spatial reasoning limitations immediately apparent even to non-experts, bridging the fundamental gap between statistical performance and intuitive assessment. LTD-Bench implements a comprehensive methodology with complementary generation tasks (testing spatial imagination) and recognition tasks (assessing spatial perception) across three progressively challenging difficulty levels, methodically evaluating both directions of the critical language-spatial mapping. Our extensive experiments with state-of-the-art models expose an alarming capability gap: even LLMs achieving impressive results on traditional benchmarks demonstrate profound deficiencies in establishing bidirectional mappings between language and spatial concept--a fundamental limitation that undermines their potential as genuine world models. Furthermore, LTD-Bench's visual outputs enable powerful diagnostic analysis, offering a potential approach to investigate model similarity.