Hybrid LLM-Embedded Dialogue Agents for Learner Reflection: Designing Responsive and Theory-Driven Interactions

TL;DR

The findings indicate that LLM-embedded dialogues supported richer learner reflections on goals and activities, but also introduced challenges due to repetitiveness and misalignment in prompts, reducing engagement.

Abstract

Dialogue systems have long supported learner reflections, with theoretically grounded, rule-based designs offering structured scaffolding but often struggling to respond to shifts in engagement. Large Language Models (LLMs), in contrast, can generate context-sensitive responses but are not informed by decades of research on how learning interactions should be structured, raising questions about their alignment with pedagogical theories. This paper presents a hybrid dialogue system that embeds LLM responsiveness within a theory-aligned, rule-based framework to support learner reflections in a culturally responsive robotics summer camp. The rule-based structure grounds dialogue in self-regulated learning theory, while the LLM decides when and how to prompt deeper reflections, responding to evolving conversation context. We analyze themes across dialogues to explore how our hybrid system shaped learner reflections. Our findings indicate that LLM-embedded dialogues supported richer learner reflections on goals and activities, but also introduced challenges due to repetitiveness and misalignment in prompts, reducing engagement.

Figures (4)

• Figure 1: This diagram depicts our hybrid dialogue system architecture modeled as a Finite State Machine and illustrates how logic flows across the rule-based prompts, learner responses, and conditional transitions at each system-learner turn.
• Figure 2: The two stages of LLM in our dialogue system: (1) Relevance Check, where a dynamic prompt with prompt-specific information and few-shot examples determines binary relevance; and (2) Contextual Generation, where, upon a NO in relevance check, a new prompt incorporating prompt-specific information, dialogue history, and few-shot examples guides the LLM in producing a follow-up.
• Figure 3: The figure shows the chat widget embedded in our system's frontend programming interface. Learners interact with the system through this widget using text or speech-to-text input, while system responses are delivered as both text and audio. The example displayed shows a dialogue snippet from our camp where the system prompts a learner to reflect on their goals and plans for the robot showcase. The learner responses "Yes" and "No" are part of a menu-based selection, and "walk and ctalk" and "by coding" are free-text responses.
• Figure 4: The rule-based prompts for the dialogue system.