SSRLBot: Designing and Developing a Large Language Model-based Agent using Socially Shared Regulated Learning

TL;DR

This paper addresses evaluating medical students' diagnostic reasoning within team-based settings using the SSRL framework. It presents SSRLBot, an LLM-powered agent that summarizes dialogue, analyzes SSRL skills, and evaluates diagnostic outcomes. Evaluated on six groups (12 participants, 1926 turns) in a BioWorld medical simulation, SSRLBot delivered theory-aligned evaluations, linked behaviors to SSRL dimensions, and achieved 100% diagnostic accuracy with actionable feedback. The work fills a gap in medical education by enabling theory-driven assessment of teamwork and SSRL, with implications for improving high-stakes clinical decision-making.

Abstract

Large language model (LLM)--based agents have emerged as pivotal tools in assisting human experts across various fields by transforming complex tasks into more efficient workflows and providing actionable stakeholder insights. Despite their potential, the application of LLM-based agents for medical education remains underexplored. The study aims to assist in evaluating the students' process and outcomes on medical case diagnosis and discussion while incorporating the theoretical framework of Socially Shared Regulation of Learning (SSRL) to assess student performance. SSRL emphasizes metacognitive, cognitive, motivational, and emotional interactions, highlighting the collaborative management of learning processes to improve decision-making outcomes. Grounded in SSRL theory, this tool paper introduces SSRLBot, an LLM-based agent designed to enable team members to reflect on their diagnostic performance and the key SSRL skills that foster team success. SSRLBot's core functions include summarizing dialogue content, analyzing participants' SSRL skills, and evaluating students' diagnostic results. Meanwhile, we evaluated SSRLBot through diagnostic conversation data collected from six groups (12 participants, 1926 conversational turns). Results showed that SSRLBot can deliver detailed, theory-aligned evaluations, link specific behaviors to SSRL dimensions, and offer actionable recommendations for improving teamwork. The findings address a critical gap in medical education, advancing the application of LLM agents to enhance team-based decision-making and collaboration in high-stakes environments.

Figures (2)

• Figure 1: The Workflow of the System Architecture for SSRLBot.
• Figure 2: The Demonstration of SSRLBot Functions.