Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating LLMs' Reasoning Over Ordered Procedural Steps

Adrita Anika, Md Messal Monem Miah

TL;DR

This work tackles the problem of procedural reasoning by reconstructing globally ordered sequences from shuffled steps in cooking recipes. The authors introduce a multi-m metric evaluation framework and apply it to zero-shot and few-shot prompting of five instruction-tuned LLMs, analyzing performance as sequence length and shuffling difficulty vary. Key findings show that models improve from 0 to 3 shots but gain little from additional demonstrations, with Qwen-3 and GPT-4o achieving the strongest overall ordering quality while longer and more disordered sequences reveal remaining gaps in precise step-level reasoning. The study provides insight into the limits of current LLMs for procedural tasks and motivates future work on dataset breadth, cross-domain generalization, and targeted fine-tuning to enhance global sequence reconstruction capabilities.

Abstract

Reasoning over procedural sequences, where the order of steps directly impacts outcomes, is a critical capability for large language models (LLMs). In this work, we study the task of reconstructing globally ordered sequences from shuffled procedural steps, using a curated dataset of food recipes, a domain where correct sequencing is essential for task success. We evaluate several LLMs under zero-shot and few-shot settings and present a comprehensive evaluation framework that adapts established metrics from ranking and sequence alignment. These include Kendall's Tau, Normalized Longest Common Subsequence (NLCS), and Normalized Edit Distance (NED), which capture complementary aspects of ordering quality. Our analysis shows that model performance declines with increasing sequence length, reflecting the added complexity of longer procedures. We also find that greater step displacement in the input, corresponding to more severe shuffling, leads to further degradation. These findings highlight the limitations of current LLMs in procedural reasoning, especially with longer and more disordered inputs.

Evaluating LLMs' Reasoning Over Ordered Procedural Steps

TL;DR

This work tackles the problem of procedural reasoning by reconstructing globally ordered sequences from shuffled steps in cooking recipes. The authors introduce a multi-m metric evaluation framework and apply it to zero-shot and few-shot prompting of five instruction-tuned LLMs, analyzing performance as sequence length and shuffling difficulty vary. Key findings show that models improve from 0 to 3 shots but gain little from additional demonstrations, with Qwen-3 and GPT-4o achieving the strongest overall ordering quality while longer and more disordered sequences reveal remaining gaps in precise step-level reasoning. The study provides insight into the limits of current LLMs for procedural tasks and motivates future work on dataset breadth, cross-domain generalization, and targeted fine-tuning to enhance global sequence reconstruction capabilities.

Abstract

Reasoning over procedural sequences, where the order of steps directly impacts outcomes, is a critical capability for large language models (LLMs). In this work, we study the task of reconstructing globally ordered sequences from shuffled procedural steps, using a curated dataset of food recipes, a domain where correct sequencing is essential for task success. We evaluate several LLMs under zero-shot and few-shot settings and present a comprehensive evaluation framework that adapts established metrics from ranking and sequence alignment. These include Kendall's Tau, Normalized Longest Common Subsequence (NLCS), and Normalized Edit Distance (NED), which capture complementary aspects of ordering quality. Our analysis shows that model performance declines with increasing sequence length, reflecting the added complexity of longer procedures. We also find that greater step displacement in the input, corresponding to more severe shuffling, leads to further degradation. These findings highlight the limitations of current LLMs in procedural reasoning, especially with longer and more disordered inputs.

Paper Structure

This paper contains 25 sections, 5 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Definition
  4. Experimental Setup
  5. Inference Settings
  6. Evaluation Metrics
  7. Step Accuracy (Acc).
  8. Kendall's Tau ($\tau$) (KTau).
  9. Normalized Edit Distance (NED).
  10. Normalized Longest Common Subsequence (NLCS).
  11. Results and Analysis
  12. Performance in Zero-Shot and Few-Shot Settings
  13. Impact of Number of Steps on Ordering Performance
  14. Impact of Step Average Positional Displacement on Model Performance
  15. Conclusion
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Example of the step ordering task. Given a shuffled list of recipe instructions (top), the goal is to recover the correct sequence (bottom) required to successfully complete the recipe. The middle row shows the gold permutation that reorders the input into the correct order.
  • Figure 2: Distribution of step movement distances across recipes of different sequence lengths.
  • Figure 3: 3-shot performance of models (Acc, NLCS, KTau, NED) across varying numbers of steps.
  • Figure 4: Smoothed 3-shot performance of models (Acc, NLCS, KTau, NED) across average positional displacement
  • Figure 5: LLM prompt used for data curation
  • ...and 2 more figures