Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Opportunities of Large Language Models for Programming Process Data

John Edwards, Arto Hellas, Juho Leinonen

TL;DR

The findings highlight the potential of combining keystroke data with LLMs to automate formative feedback, showing that the computing education research and practice community is again one step closer to automating formative programming process feedback.

Abstract

Computing educators and researchers have used programming process data to understand how programs are constructed and what sorts of problems students struggle with. Although such data shows promise for using it for feedback, fully automated programming process feedback systems have still been an under-explored area. The recent emergence of large language models (LLMs) have yielded additional opportunities for researchers in a wide variety of fields. LLMs are efficient at transforming content from one format to another, leveraging the body of knowledge they have been trained with in the process. In this article, we discuss opportunities of using LLMs for analyzing programming process data. To complement our discussion, we outline a case study where we have leveraged LLMs for automatically summarizing the programming process and for creating formative feedback on the programming process. Overall, our discussion and findings highlight that the computing education research and practice community is again one step closer to automating formative programming process-focused feedback.

On the Opportunities of Large Language Models for Programming Process Data

TL;DR

The findings highlight the potential of combining keystroke data with LLMs to automate formative feedback, showing that the computing education research and practice community is again one step closer to automating formative programming process feedback.

Abstract

Computing educators and researchers have used programming process data to understand how programs are constructed and what sorts of problems students struggle with. Although such data shows promise for using it for feedback, fully automated programming process feedback systems have still been an under-explored area. The recent emergence of large language models (LLMs) have yielded additional opportunities for researchers in a wide variety of fields. LLMs are efficient at transforming content from one format to another, leveraging the body of knowledge they have been trained with in the process. In this article, we discuss opportunities of using LLMs for analyzing programming process data. To complement our discussion, we outline a case study where we have leveraged LLMs for automatically summarizing the programming process and for creating formative feedback on the programming process. Overall, our discussion and findings highlight that the computing education research and practice community is again one step closer to automating formative programming process-focused feedback.

Paper Structure

This paper contains 15 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Programming Process Data
  3. Types of data
  4. Uses of programming process data
  5. Possibilities of Large Language Models for Programming Process Data
  6. Case Study: Analyzing Programming Process Data with LLMs
  7. Process data and preprocessing
  8. Large language models and prompt engineering
  9. Data generation
  10. Evaluation
  11. Results
  12. Surface-level analysis
  13. Analysis of programming process improvement suggestions
  14. Researcher insights from programming process summarization and feedback
  15. Discussion

Figures (1)

  • Figure 1: Prompt format for providing feedback on the programming process. The prompt included (1) a description of the context and the persona, (2) the broader task, (3) the handout, (4) the format of the input data, (5) the explicit task instructions, and (6) the process data.