Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction

Yingjie He, Zhaolu Kang, Kehan Jiang, Qianyuan Zhang, Jiachen Qian, Chunlei Meng, Yujie Feng, Yuan Wang, Jiabao Dou, Aming Wu, Leqi Zheng, Pengxiang Zhao, Jiaxin Liu, Zeyu Zhang, Lei Wang, Guansu Wang, Qishi Zhan, Xiaomin He, Meisheng Zhang, Jianyuan Ni

TL;DR

OrderProbe is introduced, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean, which have a unique canonical order and thus support exact-match scoring.

Abstract

Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored. Sentence-level restoration is ill-posed for automated evaluation because multiple valid word orders often exist. We introduce OrderProbe, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean, which have a unique canonical order and thus support exact-match scoring. We further propose a diagnostic framework that evaluates models beyond recovery accuracy, including semantic fidelity, logical validity, consistency, robustness sensitivity, and information density. Experiments on twelve widely used LLMs show that structural reconstruction remains difficult even for frontier systems: zero-shot recovery frequently falls below 35%. We also observe a consistent dissociation between semantic recall and structural planning, suggesting that structural robustness is not an automatic byproduct of semantic competence.

How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction

TL;DR

OrderProbe is introduced, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean, which have a unique canonical order and thus support exact-match scoring.

Abstract

Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored. Sentence-level restoration is ill-posed for automated evaluation because multiple valid word orders often exist. We introduce OrderProbe, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean, which have a unique canonical order and thus support exact-match scoring. We further propose a diagnostic framework that evaluates models beyond recovery accuracy, including semantic fidelity, logical validity, consistency, robustness sensitivity, and information density. Experiments on twelve widely used LLMs show that structural reconstruction remains difficult even for frontier systems: zero-shot recovery frequently falls below 35%. We also observe a consistent dissociation between semantic recall and structural planning, suggesting that structural robustness is not an automatic byproduct of semantic competence.
Paper Structure (89 sections, 19 equations, 10 figures, 11 tables)

This paper contains 89 sections, 19 equations, 10 figures, 11 tables.

Table of Contents

  1. Introduction
  2. OrderProbe
  3. Stage I: Multi-source Collection
  4. Stage II: Expert Filtering
  5. Stage III: Semantic Reference Construction
  6. Stage IV: Permutation-based Perturbation Generation
  7. Dataset Statistics
  8. Evaluation Framework
  9. Evaluation Scope.
  10. Primary Metric: Recovery Rate
  11. Diagnostic Metrics (Beyond Recovery)
  12. Semantic Fidelity ($S^{mean}_{Acc}$).
  13. Logical Validity ($S_{Logic}$).
  14. Structural Consistency ($S_{Cons}$).
  15. Robustness ($S_{Rob}$).
  16. ...and 74 more sections

Figures (10)

  • Figure 1: OrderProbe example illustrating semantic--structure dissociation. Given a scrambled four-character expression, humans can often recover the unique canonical order using semantic anchors and structural priors. Many LLMs, however, produce correct meanings while failing exact reconstruction, revealing a gap between semantic recall and structural planning.
  • Figure 2: OrderProbe construction pipeline. We collect raw four-character expressions from multiple sources, apply expert filtering to obtain canonical roots, build semantic references via dictionary grounding and verified augmentation, and generate the full non-identity permutation space as structurally perturbed evaluation inputs.
  • Figure 3: Overview of OrderProbe. The chart displays the distribution of script typologies and syntactic structures across the 3543 samples.
  • Figure 4: Performance Comparison across 23 Permutation Patterns. The chart illustrates recovery rates under baseline, few shot, and chain of thought settings, revealing the impact of anchor displacement on model stability.
  • Figure 5: Performance by syntactic category. Parallel structures are easiest due to redundancy and symmetry cues, while asymmetric dependency patterns remain harder for reconstruction. Metrics are reported to diagnose whether failures arise from semantic gaps, instability, or structural sensitivity.
  • ...and 5 more figures