Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Procedural Skill Explanations via Constrained Generation: A Symbolic-LLM Hybrid Architecture

Rahul Dass, Thomas Bowlin, Zebing Li, Xiao Jin, Ashok Goel

TL;DR

This work tackles the gap in procedural skill explanations produced by LLMs, which often lack explicit causal, teleological, and decompositional structure. It introduces Ivy, a two-layer AI coaching system that constrains LLM generation with TMK-Structured symbolic models—encoding causal transitions via finite-state machines, goal hierarchies, and hierarchical decomposition—to produce structured, multi-step explanations. A four-stage pipeline (scope classification, TMK retrieval, constrained synthesis, coherence optimization) enforces procedural grounding and reduces hallucinations, with empirical evaluation showing improved correctness and inferential structure, especially in decomposition. The approach demonstrates a scalable, education-focused paradigm that blends symbolic grounding with neural synthesis to deliver pedagogically faithful explanations in intelligent coaching systems.

Abstract

In procedural skill learning, instructional explanations must convey not just steps, but the causal, goal-directed, and compositional logic behind them. Large language models (LLMs) often produce fluent yet shallow responses that miss this structure. We present Ivy, an AI coaching system that delivers structured, multi-step explanations by combining symbolic Task-Method-Knowledge (TMK) models with a generative interpretation layer-an LLM that constructs explanations while being constrained by TMK structure. TMK encodes causal transitions, goal hierarchies, and problem decompositions, and guides the LLM within explicit structural bounds. We evaluate Ivy against responses against GPT and retrieval-augmented GPT baselines using expert and independent annotations across three inferential dimensions. Results show that symbolic constraints consistently improve the structural quality of explanations for "how" and "why" questions. This study demonstrates a scalable AI for education approach that strengthens the pedagogical value of AI-generated explanations in intelligent coaching systems.

Improving Procedural Skill Explanations via Constrained Generation: A Symbolic-LLM Hybrid Architecture

TL;DR

This work tackles the gap in procedural skill explanations produced by LLMs, which often lack explicit causal, teleological, and decompositional structure. It introduces Ivy, a two-layer AI coaching system that constrains LLM generation with TMK-Structured symbolic models—encoding causal transitions via finite-state machines, goal hierarchies, and hierarchical decomposition—to produce structured, multi-step explanations. A four-stage pipeline (scope classification, TMK retrieval, constrained synthesis, coherence optimization) enforces procedural grounding and reduces hallucinations, with empirical evaluation showing improved correctness and inferential structure, especially in decomposition. The approach demonstrates a scalable, education-focused paradigm that blends symbolic grounding with neural synthesis to deliver pedagogically faithful explanations in intelligent coaching systems.

Abstract

In procedural skill learning, instructional explanations must convey not just steps, but the causal, goal-directed, and compositional logic behind them. Large language models (LLMs) often produce fluent yet shallow responses that miss this structure. We present Ivy, an AI coaching system that delivers structured, multi-step explanations by combining symbolic Task-Method-Knowledge (TMK) models with a generative interpretation layer-an LLM that constructs explanations while being constrained by TMK structure. TMK encodes causal transitions, goal hierarchies, and problem decompositions, and guides the LLM within explicit structural bounds. We evaluate Ivy against responses against GPT and retrieval-augmented GPT baselines using expert and independent annotations across three inferential dimensions. Results show that symbolic constraints consistently improve the structural quality of explanations for "how" and "why" questions. This study demonstrates a scalable AI for education approach that strengthens the pedagogical value of AI-generated explanations in intelligent coaching systems.

Paper Structure

This paper contains 38 sections, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Structured Reasoning and Transfer
  4. Constrained Generation for Multi-Step Inferencing
  5. Hybrid Symbolic-Neural Architectures for Reducing Hallucinations
  6. Constrained Generation Architecture
  7. Two-layered Inferencing Architecture
  8. Four-Stage Pipeline and Algorithmic Implementation
  9. Stage 1: Question Scope Classification.
  10. Stage 2: Structured Knowledge Retrieval.
  11. Stage 3: Constrained Synthesis.
  12. Stage 4: Coherence Optimization.
  13. Structured Constraints And Inferential Quality: A System-Level Evaluation
  14. Discussion
  15. Expert-Judged Correctness
  16. ...and 23 more sections

Figures (3)

  • Figure 1: Concept diagram of Ivy's two-layered inferencing: TMK encodes structured knowledge; LLMs perform runtime inference within these constraints.
  • Figure 2: A finite state machine representation of the ReturnGuardMechanism from the TMK-Structured model of the Guards and Prisoners problem, illustrating causal transitions (solid arrows) and failure branches (dashed) to represent procedural logic in TMK Mechanisms.
  • Figure 3: A four-stage constrained generation architecture for AI coaching systems. Instantiated using Ivy, an AI coach integrated within Ed Lessons to support procedural skill learning, the architecture demonstrates how an AI coach leverages symbolic control via TMK models to generate responses.