Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging Learnersourcing and AI: Exploring the Dynamics of Student-AI Collaborative Feedback Generation

Anjali Singh, Christopher Brooks, Xu Wang, Warren Li, Juho Kim, Deepti Pandey

TL;DR

This study addresses how to optimize formative feedback in data science by pairing learnersourcing with AI in a student-AI collaborative workflow. Using a randomized crossover design with 72 master's students, it compares independent hint-writing to revising GPT-4 hints for incorrect programming solutions, assessing hint quality across attributes like accuracy, specificity, and utility, along with time spent and learner perceptions. The findings indicate that AI-assisted revisions can elevate hint quality and perceived usefulness, especially when GPT-4 hints are accurate, though effects vary with task difficulty and student performance, and learner trust and design choices influence adoption. The work highlights practical design implications for scaling feedback with AI while preserving the cognitive benefits of learnersourcing, and suggests future avenues for selective or on-demand AI scaffolds to optimize learning outcomes in data science education.

Abstract

This paper explores the space of optimizing feedback mechanisms in complex domains, such as data science, by combining two prevailing approaches: Artificial Intelligence (AI) and learnersourcing. Towards addressing the challenges posed by each approach, this work compares traditional learnersourcing with an AI-supported approach. We report on the results of a randomized controlled experiment conducted with 72 Master's level students in a data visualization course, comparing two conditions: students writing hints independently versus revising hints generated by GPT-4. The study aimed to evaluate the quality of learnersourced hints, examine the impact of student performance on hint quality, gauge learner preference for writing hints with or without AI support, and explore the potential of the student-AI collaborative exercise in fostering critical thinking about LLMs. Based on our findings, we provide insights for designing learnersourcing activities leveraging AI support and optimizing students' learning as they interact with LLMs.

Bridging Learnersourcing and AI: Exploring the Dynamics of Student-AI Collaborative Feedback Generation

TL;DR

This study addresses how to optimize formative feedback in data science by pairing learnersourcing with AI in a student-AI collaborative workflow. Using a randomized crossover design with 72 master's students, it compares independent hint-writing to revising GPT-4 hints for incorrect programming solutions, assessing hint quality across attributes like accuracy, specificity, and utility, along with time spent and learner perceptions. The findings indicate that AI-assisted revisions can elevate hint quality and perceived usefulness, especially when GPT-4 hints are accurate, though effects vary with task difficulty and student performance, and learner trust and design choices influence adoption. The work highlights practical design implications for scaling feedback with AI while preserving the cognitive benefits of learnersourcing, and suggests future avenues for selective or on-demand AI scaffolds to optimize learning outcomes in data science education.

Abstract

This paper explores the space of optimizing feedback mechanisms in complex domains, such as data science, by combining two prevailing approaches: Artificial Intelligence (AI) and learnersourcing. Towards addressing the challenges posed by each approach, this work compares traditional learnersourcing with an AI-supported approach. We report on the results of a randomized controlled experiment conducted with 72 Master's level students in a data visualization course, comparing two conditions: students writing hints independently versus revising hints generated by GPT-4. The study aimed to evaluate the quality of learnersourced hints, examine the impact of student performance on hint quality, gauge learner preference for writing hints with or without AI support, and explore the potential of the student-AI collaborative exercise in fostering critical thinking about LLMs. Based on our findings, we provide insights for designing learnersourcing activities leveraging AI support and optimizing students' learning as they interact with LLMs.
Paper Structure (17 sections, 2 figures, 1 table)

This paper contains 17 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reflecting on Mistakes as a Learning Activity
  4. Learnersourcing and AI for Generating Educational Content
  5. Method
  6. Course Context and Study Population
  7. Experiment Design
  8. Reflection Assignments
  9. Prompting GPT-4 for Hints
  10. Data Collection and Outcome Measures
  11. Understanding Students' Learning Experience
  12. Data Analysis
  13. Results
  14. Data Overview
  15. Study Results
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Normalized counts of students per each level of Utility (with '7' representing most usable and '1' representing least usable hints) for each reflection. The yellow vertical lines indicate the Utility of the GPT-4 hint for each reflection.
  • Figure 2: Students' responses regarding the extent to which the AI-hint-revision variant of the exercise helped them think critically about the accuracy and appropriateness of GPT-4 responses