Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tutorly: Turning Programming Videos Into Apprenticeship Learning Environments with LLMs

Wengxi Li, Roy Pea, Nick Haber, Hari Subramonyam

TL;DR

Tutorly introduces an LLM-powered apprenticeship environment that turns programming videos into guided, goal-driven learning experiences inside JupyterLab. By segmenting videos into learning goals, applying CogApp moves through a structured prompt pipeline, and updating a Bayesian Knowledge Tracing-based student model, Tutorly delivers targeted, interactive tutoring aligned with learner progress. Technical evaluation shows robust segmentation and high controllability of prompts, while a within-subject user study reports improved learning outcomes and engagement, though latency and video segmentation length remain challenges. Overall, Tutorly demonstrates the practical potential of integrating cognitive apprenticeship pedagogy with large-language models to enhance video-based programming education in an extensible notebook workflow.

Abstract

Online programming videos, including tutorials and streamcasts, are widely popular and contain a wealth of expert knowledge. However, effectively utilizing these resources to achieve targeted learning goals can be challenging. Unlike direct tutoring, video content lacks tailored guidance based on individual learning paces, personalized feedback, and interactive engagement necessary for support and monitoring. Our work transforms programming videos into one-on-one tutoring experiences using the cognitive apprenticeship framework. Tutorly, developed as a JupyterLab Plugin, allows learners to (1) set personalized learning goals, (2) engage in learning-by-doing through a conversational LLM-based mentor agent, (3) receive guidance and feedback based on a student model that steers the mentor moves. In a within-subject study with 16 participants learning exploratory data analysis from a streamcast, Tutorly significantly improved their performance from 61.9% to 76.6% based on a post-test questionnaire. Tutorly demonstrates the potential for enhancing programming video learning experiences with LLM and learner modeling.

Tutorly: Turning Programming Videos Into Apprenticeship Learning Environments with LLMs

TL;DR

Tutorly introduces an LLM-powered apprenticeship environment that turns programming videos into guided, goal-driven learning experiences inside JupyterLab. By segmenting videos into learning goals, applying CogApp moves through a structured prompt pipeline, and updating a Bayesian Knowledge Tracing-based student model, Tutorly delivers targeted, interactive tutoring aligned with learner progress. Technical evaluation shows robust segmentation and high controllability of prompts, while a within-subject user study reports improved learning outcomes and engagement, though latency and video segmentation length remain challenges. Overall, Tutorly demonstrates the practical potential of integrating cognitive apprenticeship pedagogy with large-language models to enhance video-based programming education in an extensible notebook workflow.

Abstract

Online programming videos, including tutorials and streamcasts, are widely popular and contain a wealth of expert knowledge. However, effectively utilizing these resources to achieve targeted learning goals can be challenging. Unlike direct tutoring, video content lacks tailored guidance based on individual learning paces, personalized feedback, and interactive engagement necessary for support and monitoring. Our work transforms programming videos into one-on-one tutoring experiences using the cognitive apprenticeship framework. Tutorly, developed as a JupyterLab Plugin, allows learners to (1) set personalized learning goals, (2) engage in learning-by-doing through a conversational LLM-based mentor agent, (3) receive guidance and feedback based on a student model that steers the mentor moves. In a within-subject study with 16 participants learning exploratory data analysis from a streamcast, Tutorly significantly improved their performance from 61.9% to 76.6% based on a post-test questionnaire. Tutorly demonstrates the potential for enhancing programming video learning experiences with LLM and learner modeling.
Paper Structure (53 sections, 1 equation, 5 figures, 5 tables, 4 algorithms)

This paper contains 53 sections, 1 equation, 5 figures, 5 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video-based Learning
  4. Cognitive Apprenticeship in Computer Science Education
  5. Conversational Intelligent Tutoring Systems
  6. User Experience
  7. Video Context
  8. Apprenticeship Learning with Tutorly
  9. System Architecture
  10. Video Segmentation by Learning Goals
  11. Instructional Prompt Generator
  12. Summarize the knowledge in each video segment.
  13. Select Pedagogical Strategy for each piece of knowledge.
  14. Generating a DSL for multi-turn conversation.
  15. Student Behavior Monitoring
  16. ...and 38 more sections

Figures (5)

  • Figure 1: Tutorly user interface as a JupyterLab Extension. (a) Code panel with cell structure, (b) Tutorly Chat Interface with Rich Conversational Features
  • Figure 2: Overview of Tutorly's System Architecture. From Left to Right: The video Transcript is segmented based on learning goals, and then for each segment, the knowledge is summarized based on knowledge type. Next the appropriate CogApp mentor move is selected for the knowledge and using the student model. Finally a DSL representation is produced to prompt GPT for generating the conversation.
  • Figure 3: Tutorly Prompt Configuration and Testing Utility
  • Figure 4: Participant Responses to Post-Study System Usability Questionnaire.
  • Figure 5: The six cognitive apprenticeship methods and an example message of each method.