Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT

TL;DR

The paper investigates how novice programmers interact with ChatGPT-3.5 during introductory programming tasks by analyzing authentic chat protocols from 213 students (2335 prompts) in a university course. It employs quantitative metrics and qualitative coding to characterize prompt types, follow-up interactions, and problem-solving steps, revealing three distinct usage patterns and two prompting patterns. The findings have practical implications for designing classroom guidance and guardrails to foster critical, effective use of GenAI tools in early computing education. The study also highlights limitations and calls for replication and exploration of specialized coding AI to augment learning outcomes.

Abstract

Large Language Models (LLMs) have taken the world by storm, and students are assumed to use related tools at a great scale. In this research paper we aim to gain an understanding of how introductory programming students chat with LLMs and related tools, e.g., ChatGPT-3.5. To address this goal, computing students at a large German university were motivated to solve programming exercises with the assistance of ChatGPT as part of their weekly introductory course exercises. Then students (n=213) submitted their chat protocols (with 2335 prompts in sum) as data basis for this analysis. The data was analyzed w.r.t. the prompts, frequencies, the chats' progress, contents, and other use pattern, which revealed a great variety of interactions, both potentially supportive and concerning. Learning about students' interactions with ChatGPT will help inform and align teaching practices and instructions for future introductory programming courses in higher education.

Figures (1)

• Figure 1: Excerpts of two example chat protocols illustrating different interaction patterns (translated into English by the authors).