Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMSR@XLLM25: An Empirical Study of LLM for Structural Reasoning

Xinye Li, Mingqi Wan, Dianbo Sui

TL;DR

This work investigates verifiable, structural reasoning with large language models on the LLMSR@XLLM25 task, which requires extracting explicit problem conditions, decomposing reasoning into statement–evidence pairs, and verifying entailment at the step level. The authors implement a lightweight, non-finetuned solution using Meta-Llama-3-8B-Instruct, guided by a carefully crafted few-shot, multi-turn prompt and a minimal post-processing stage that enforces a formal JSON schema. Despite the modest resource footprint, the approach ranks 5th overall, demonstrating that disciplined prompt design and constrained reasoning can rival more complex pipelines. The study identifies verification as the main bottleneck and points to future work on stronger verifiers and lightweight entailment modules, along with process reward modeling, to enhance accuracy and transparency in LLM-driven reasoning.

Abstract

We present Team asdfo123's submission to the LLMSR@XLLM25 shared task, which evaluates large language models on producing fine-grained, controllable, and interpretable reasoning processes. Systems must extract all problem conditions, decompose a chain of thought into statement-evidence pairs, and verify the logical validity of each pair. Leveraging only the off-the-shelf Meta-Llama-3-8B-Instruct, we craft a concise few-shot, multi-turn prompt that first enumerates all conditions and then guides the model to label, cite, and adjudicate every reasoning step. A lightweight post-processor based on regular expressions normalises spans and enforces the official JSON schema. Without fine-tuning, external retrieval, or ensembling, our method ranks 5th overall, achieving macro F1 scores on par with substantially more complex and resource-consuming pipelines. We conclude by analysing the strengths and limitations of our approach and outlining directions for future research in structural reasoning with LLMs. Our code is available at https://github.com/asdfo123/LLMSR-asdfo123.

LLMSR@XLLM25: An Empirical Study of LLM for Structural Reasoning

TL;DR

This work investigates verifiable, structural reasoning with large language models on the LLMSR@XLLM25 task, which requires extracting explicit problem conditions, decomposing reasoning into statement–evidence pairs, and verifying entailment at the step level. The authors implement a lightweight, non-finetuned solution using Meta-Llama-3-8B-Instruct, guided by a carefully crafted few-shot, multi-turn prompt and a minimal post-processing stage that enforces a formal JSON schema. Despite the modest resource footprint, the approach ranks 5th overall, demonstrating that disciplined prompt design and constrained reasoning can rival more complex pipelines. The study identifies verification as the main bottleneck and points to future work on stronger verifiers and lightweight entailment modules, along with process reward modeling, to enhance accuracy and transparency in LLM-driven reasoning.

Abstract

We present Team asdfo123's submission to the LLMSR@XLLM25 shared task, which evaluates large language models on producing fine-grained, controllable, and interpretable reasoning processes. Systems must extract all problem conditions, decompose a chain of thought into statement-evidence pairs, and verify the logical validity of each pair. Leveraging only the off-the-shelf Meta-Llama-3-8B-Instruct, we craft a concise few-shot, multi-turn prompt that first enumerates all conditions and then guides the model to label, cite, and adjudicate every reasoning step. A lightweight post-processor based on regular expressions normalises spans and enforces the official JSON schema. Without fine-tuning, external retrieval, or ensembling, our method ranks 5th overall, achieving macro F1 scores on par with substantially more complex and resource-consuming pipelines. We conclude by analysing the strengths and limitations of our approach and outlining directions for future research in structural reasoning with LLMs. Our code is available at https://github.com/asdfo123/LLMSR-asdfo123.
Paper Structure (23 sections, 1 figure, 3 tables)

This paper contains 23 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Chain-of-Thought Prompting
  4. Parsing
  5. Process Reward Model
  6. Methodology
  7. Pipeline Overview
  8. Prompt Engineering
  9. Few-shot demonstrations.
  10. Three-turn template.
  11. Robust JSON Output
  12. Post-processing
  13. Efficiency Rationale
  14. Experiments
  15. Phase 1: Selecting the Question‐Parsing Shot Count
  16. ...and 8 more sections

Figures (1)

  • Figure 1: Illustration of the three-stage LLM-SR Task. (In our implementation, Verification is executed within the CP stage.)