Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reflecting in the Reflection: Integrating a Socratic Questioning Framework into Automated AI-Based Question Generation

Ondřej Holub, Essi Ryymin, Rodrigo Alves

TL;DR

The paper tackles the challenge of scalable, high-quality reflection prompts by introducing a reflection-in-reflection framework in which two role-specific agents engage in a Socratic multi-turn dialogue to refine a teacher-specified topic into a single reflective question. The approach emphasizes adaptive dialogue termination and contextual grounding (student level and materials) to improve prompt clarity, relevance, and depth. Empirical results in authentic lower-secondary ICT contexts show that dynamic stopping outperforms fixed iterations and that the two-agent protocol yields superior reflective prompts compared to a one-shot baseline. The work offers a practical, scalable tool for teacher planning, professional development, and student metacognition, with implications for integrating AI-assisted prompt design into classroom practice, while noting the need for human-in-the-loop validation in future work.

Abstract

Designing good reflection questions is pedagogically important but time-consuming and unevenly supported across teachers. This paper introduces a reflection-in-reflection framework for automated generation of reflection questions with large language models (LLMs). Our approach coordinates two role-specialized agents, a Student-Teacher and a Teacher-Educator, that engage in a Socratic multi-turn dialogue to iteratively refine a single question given a teacher-specified topic, key concepts, student level, and optional instructional materials. The Student-Teacher proposes candidate questions with brief rationales, while the Teacher-Educator evaluates them along clarity, depth, relevance, engagement, and conceptual interconnections, responding only with targeted coaching questions or a fixed signal to stop the dialogue. We evaluate the framework in an authentic lower-secondary ICT setting on the topic, using GPT-4o-mini as the backbone model and a stronger GPT- 4-class LLM as an external evaluator in pairwise comparisons of clarity, relevance, depth, and overall quality. First, we study how interaction design and context (dynamic vs.fixed iteration counts; presence or absence of student level and materials) affect question quality. Dynamic stopping combined with contextual information consistently outperforms fixed 5- or 10-step refinement, with very long dialogues prone to drift or over-complication. Second, we show that our two-agent protocol produces questions that are judged substantially more relevant and deeper, and better overall, than a one-shot baseline using the same backbone model.

Reflecting in the Reflection: Integrating a Socratic Questioning Framework into Automated AI-Based Question Generation

TL;DR

The paper tackles the challenge of scalable, high-quality reflection prompts by introducing a reflection-in-reflection framework in which two role-specific agents engage in a Socratic multi-turn dialogue to refine a teacher-specified topic into a single reflective question. The approach emphasizes adaptive dialogue termination and contextual grounding (student level and materials) to improve prompt clarity, relevance, and depth. Empirical results in authentic lower-secondary ICT contexts show that dynamic stopping outperforms fixed iterations and that the two-agent protocol yields superior reflective prompts compared to a one-shot baseline. The work offers a practical, scalable tool for teacher planning, professional development, and student metacognition, with implications for integrating AI-assisted prompt design into classroom practice, while noting the need for human-in-the-loop validation in future work.

Abstract

Designing good reflection questions is pedagogically important but time-consuming and unevenly supported across teachers. This paper introduces a reflection-in-reflection framework for automated generation of reflection questions with large language models (LLMs). Our approach coordinates two role-specialized agents, a Student-Teacher and a Teacher-Educator, that engage in a Socratic multi-turn dialogue to iteratively refine a single question given a teacher-specified topic, key concepts, student level, and optional instructional materials. The Student-Teacher proposes candidate questions with brief rationales, while the Teacher-Educator evaluates them along clarity, depth, relevance, engagement, and conceptual interconnections, responding only with targeted coaching questions or a fixed signal to stop the dialogue. We evaluate the framework in an authentic lower-secondary ICT setting on the topic, using GPT-4o-mini as the backbone model and a stronger GPT- 4-class LLM as an external evaluator in pairwise comparisons of clarity, relevance, depth, and overall quality. First, we study how interaction design and context (dynamic vs.fixed iteration counts; presence or absence of student level and materials) affect question quality. Dynamic stopping combined with contextual information consistently outperforms fixed 5- or 10-step refinement, with very long dialogues prone to drift or over-complication. Second, we show that our two-agent protocol produces questions that are judged substantially more relevant and deeper, and better overall, than a one-shot baseline using the same backbone model.
Paper Structure (21 sections, 7 equations, 6 figures, 1 algorithm)

This paper contains 21 sections, 7 equations, 6 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Reflection
  4. Educational Models for Reflection
  5. Automated Question Generation
  6. Methodology
  7. Dialogue Dynamics
  8. Student--Teacher Agent
  9. Teacher--Educator Agent
  10. Experiments
  11. RQ1 (Role of Context and Interaction Design).
  12. RQ2 (Protocol Effectiveness).
  13. Domain and Auxiliary Data
  14. Evaluation Protocol
  15. RQ1: Role of Context and Interaction Design
  16. ...and 6 more sections

Figures (6)

  • Figure 1: Reflection-in-reflection question-generation architecture. A Human Teacher provides topic & concepts (and optional materials) to the Question Generation System. Inside the system, a Student–Teacher LLM drafts reflection questions while a Teacher–Educator LLM applies a Socratic framework to pose guiding questions; their iterative loop refines the draft. The system returns the finalized reflection question to the instructor.
  • Figure 2: Heatmap evaluation of the Clarity component of the generated questions. Rows represent configurations $\alpha$ and columns represent configurations $\beta$. Each entry shows the score $\gamma(\alpha,\beta)$. Legend:IT: Iteration Type; L: Student Level; M: Presence of Supporting Materials; DYN: Dynamic number of iterations; F05: Fixed to 5 iterations; F10: Fixed to 10 iterations.
  • Figure 3: Heatmap evaluation of the Relevance component of the generated questions. Rows represent configurations $\alpha$ and columns represent configurations $\beta$. Each entry shows the score $\gamma(\alpha,\beta)$. Legend:IT: Iteration Type; L: Student Level; M: Presence of Supporting Materials; DYN: Dynamic number of iterations; F05: Fixed to 5 iterations; F10: Fixed to 10 iterations.
  • Figure 4: Heatmap evaluation of the Depth component of the generated questions. Rows represent configurations $\alpha$ and columns represent configurations $\beta$. Each entry shows the score $\gamma(\alpha,\beta)$. Legend:IT: Iteration Type; L: Student Level; M: Presence of Supporting Materials; DYN: Dynamic number of iterations; F05: Fixed to 5 iterations; F10: Fixed to 10 iterations.
  • Figure 5: Heatmap evaluation of the Overall Quality of the generated questions. Rows represent configurations $\alpha$ and columns represent configurations $\beta$. Each entry shows the score $\gamma(\alpha,\beta)$. Legend:IT: Iteration Type; L: Student Level; M: Presence of Supporting Materials; DYN: Dynamic number of iterations; F05: Fixed to 5 iterations; F10: Fixed to 10 iterations.
  • ...and 1 more figures