Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating Situated Reflection Triggers about Alternative Solution Paths: A Case Study of Generative AI for Computer-Supported Collaborative Learning

Atharva Naik, Jessica Ruhan Yin, Anusha Kamath, Qianou Ma, Sherry Tongshuang Wu, Charles Murray, Christopher Bogart, Majd Sakr, Carolyn P. Rose

TL;DR

The study investigates context-aware reflection triggers delivered via GenAI during a collaborative SQL optimization task in CSCL, using mob programming and an OpenAI Reflection Generator embedded in an Online Programming Exercise bot. It introduces a four-component reflection pipeline—triggering, personalization, validation, and scheduling—and demonstrates a prompt-design approach and learning-resource enhancements. In a pilot with 34 students, tailored reflections influenced engagement and task progression but did not yield statistically significant learning gains over baseline, partially due to pretest imbalances and engagement issues. The work demonstrates feasibility of situated reflection in collaborative CS education and outlines concrete avenues to improve content relevance, context integration, and evaluation sensitivity for future deployments.

Abstract

An advantage of Large Language Models (LLMs) is their contextualization capability - providing different responses based on student inputs like solution strategy or prior discussion, to potentially better engage students than standard feedback. We present a design and evaluation of a proof-of-concept LLM application to offer students dynamic and contextualized feedback. Specifically, we augment an Online Programming Exercise bot for a college-level Cloud Computing course with ChatGPT, which offers students contextualized reflection triggers during a collaborative query optimization task in database design. We demonstrate that LLMs can be used to generate highly situated reflection triggers that incorporate details of the collaborative discussion happening in context. We discuss in depth the exploration of the design space of the triggers and their correspondence with the learning objectives as well as the impact on student learning in a pilot study with 34 students.

Generating Situated Reflection Triggers about Alternative Solution Paths: A Case Study of Generative AI for Computer-Supported Collaborative Learning

TL;DR

The study investigates context-aware reflection triggers delivered via GenAI during a collaborative SQL optimization task in CSCL, using mob programming and an OpenAI Reflection Generator embedded in an Online Programming Exercise bot. It introduces a four-component reflection pipeline—triggering, personalization, validation, and scheduling—and demonstrates a prompt-design approach and learning-resource enhancements. In a pilot with 34 students, tailored reflections influenced engagement and task progression but did not yield statistically significant learning gains over baseline, partially due to pretest imbalances and engagement issues. The work demonstrates feasibility of situated reflection in collaborative CS education and outlines concrete avenues to improve content relevance, context integration, and evaluation sensitivity for future deployments.

Abstract

An advantage of Large Language Models (LLMs) is their contextualization capability - providing different responses based on student inputs like solution strategy or prior discussion, to potentially better engage students than standard feedback. We present a design and evaluation of a proof-of-concept LLM application to offer students dynamic and contextualized feedback. Specifically, we augment an Online Programming Exercise bot for a college-level Cloud Computing course with ChatGPT, which offers students contextualized reflection triggers during a collaborative query optimization task in database design. We demonstrate that LLMs can be used to generate highly situated reflection triggers that incorporate details of the collaborative discussion happening in context. We discuss in depth the exploration of the design space of the triggers and their correspondence with the learning objectives as well as the impact on student learning in a pilot study with 34 students.
Paper Structure (20 sections, 4 figures, 3 tables, 1 algorithm)

This paper contains 20 sections, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative AI in Education
  4. Intelligent Support for Collaboration
  5. Intelligent Support for Programming in CS Education
  6. Learning Activity Design
  7. Learning Task: SQL Optimization
  8. Knowledge Resources: Primers and Learning Objectives
  9. Collaborative Learning Paradigm: Mob Programming
  10. Method
  11. Reflection Design and Triggering
  12. Reflection Personalization
  13. Reflection Validation
  14. Reflection Scheduling
  15. Experimental Procedure and Study Design
  16. ...and 5 more sections

Figures (4)

  • Figure 1: An example of the COMPOSITE_IND_COL_ORDER reflection trigger along with a student response, demonstrating that our interventions make students think about the tradeoffs involved (underlined text) in the optimization.
  • Figure 2: The effect of reflection scheduling parameter $\tau$ on the range of time intervals $\Delta$ between consecutive reflections. Each point of the x-axis denotes a range of time intervals between consecutive reflections and the y-axis captures the number of reflections spaced apart by time interval $\Delta$ lying in that range.
  • Figure 3: The existing cloud infrastructure for the Online Programming Exercise (OPE) along with our newly added components (highlighted in red) for generating dynamic and personalized reflection triggers with ChatGPT
  • Figure 4: Distribution of (a, left) pre-test and (b, right) post-test scores