Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging

Priyanka Kargupta, Ishika Agarwal, Dilek Hakkani-Tur, Jiawei Han

TL;DR

TreeInstruct, an Instructor agent guided by a novel state space-based planning algorithm, is introduced, proving it to be a more effective instructor than baselines in the code debugging domain.

Abstract

Socratic questioning is an effective teaching strategy, encouraging critical thinking and problem-solving. The conversational capabilities of large language models (LLMs) show great potential for providing scalable, real-time student guidance. However, current LLMs often give away solutions directly, making them ineffective instructors. We tackle this issue in the code debugging domain with TreeInstruct, an Instructor agent guided by a novel state space-based planning algorithm. TreeInstruct asks probing questions to help students independently identify and resolve errors. It estimates a student's conceptual and syntactical knowledge to dynamically construct a question tree based on their responses and current knowledge state, effectively addressing both independent and dependent mistakes concurrently in a multi-turn interaction setting. In addition to using an existing single-bug debugging benchmark, we construct a more challenging multi-bug dataset of 150 coding problems, incorrect solutions, and bug fixes -- all carefully constructed and annotated by experts. Extensive evaluation shows TreeInstruct's state-of-the-art performance on both datasets, proving it to be a more effective instructor than baselines. Furthermore, a real-world case study with five students of varying skill levels further demonstrates TreeInstruct's ability to guide students to debug their code efficiently with minimal turns and highly Socratic questioning.

Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging

TL;DR

TreeInstruct, an Instructor agent guided by a novel state space-based planning algorithm, is introduced, proving it to be a more effective instructor than baselines in the code debugging domain.

Abstract

Socratic questioning is an effective teaching strategy, encouraging critical thinking and problem-solving. The conversational capabilities of large language models (LLMs) show great potential for providing scalable, real-time student guidance. However, current LLMs often give away solutions directly, making them ineffective instructors. We tackle this issue in the code debugging domain with TreeInstruct, an Instructor agent guided by a novel state space-based planning algorithm. TreeInstruct asks probing questions to help students independently identify and resolve errors. It estimates a student's conceptual and syntactical knowledge to dynamically construct a question tree based on their responses and current knowledge state, effectively addressing both independent and dependent mistakes concurrently in a multi-turn interaction setting. In addition to using an existing single-bug debugging benchmark, we construct a more challenging multi-bug dataset of 150 coding problems, incorrect solutions, and bug fixes -- all carefully constructed and annotated by experts. Extensive evaluation shows TreeInstruct's state-of-the-art performance on both datasets, proving it to be a more effective instructor than baselines. Furthermore, a real-world case study with five students of varying skill levels further demonstrates TreeInstruct's ability to guide students to debug their code efficiently with minimal turns and highly Socratic questioning.
Paper Structure (47 sections, 3 figures, 23 tables)

This paper contains 47 sections, 3 figures, 23 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Knowledge Tracing
  4. Socratic Reasoning in Educational AI
  5. LLMs for Interactive Education
  6. Methodology
  7. Preliminaries
  8. Problem Description
  9. Agents
  10. State Space Estimation
  11. Tree-Based Questioning
  12. Conditional generation of sibling questions.
  13. Conditional generation of child questions.
  14. Adaptive Conversation Restructuring
  15. Experiments
  16. ...and 32 more sections

Figures (3)

  • Figure 1: The Instructor's goal is to generate multi-turn Socratic questions while guiding the Student towards the correct solution.
  • Figure 2: We propose TreeInstruct, a novel tree-guided instructional questioning framework for meaningful educational debugging guidance.
  • Figure 3: We detail the process for tree-based question generation. Blue and orange text/backgrounds indicate that the Instructor and Verifier are performing the task, respectively.