Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating Large Language Models and Evaluating Student Outcomes in an Introductory Computer Science Course

Annapurna Vadaparty, David H. Smith, Samvrit Srinath, Mounika Padala, Christine Alvarado, Jamie Gorson Benario, Daniel Zingaro, Leo Porter

TL;DR

The paper addresses how integrating GenAI into an introductory CS course affects student learning and outcomes. It designs a CS1-LLM course that embeds GenAI tools like Copilot, establishes targeted design principles and learning objectives, and evaluates outcomes against international CS1 benchmarks as well as demographic groups. Key findings show exam performance largely matches historical CS1 norms, while large, open-ended projects with GenAI usage help avoid achievement gaps across demographics; students report broad usefulness of Copilot but express concerns about overreliance and uneven access during assessments. The work provides practical guidance for educators and course designers on implementing GenAI-enabled CS1, balancing foundational knowledge with AI-assisted practices and aligning assessments with industry practices.

Abstract

Generative AI (GenAI) models have broad implications for education in general, impacting the foundations of what we teach and how we assess. This is especially true in computing, where LLMs tuned for coding have demonstrated shockingly good performance on the types of assignments historically used in introductory CS (CS1) courses. As a result, CS1 courses will need to change what skills are taught and how they are assessed. Computing education researchers have begun to study student use of LLMs, but there remains much to be understood about the ways that these tools affect student outcomes. In this paper, we present the design and evaluation of a new CS1 course at a large research-intensive university that integrates the use of LLMs as a learning tool for students. We describe the design principles used to create our new CS1-LLM course, our new course objectives, and evaluation of student outcomes and perceptions throughout the course as measured by assessment scores and surveys. Our findings suggest that 1) student exam performance outcomes, including differences among demographic groups, are largely similar to historical outcomes for courses without integration of LLM tools, 2) large, open-ended projects may be particularly valuable in an LLM context, and 3) students predominantly found the LLM tools helpful, although some had concerns regarding over-reliance on the tools.

Integrating Large Language Models and Evaluating Student Outcomes in an Introductory Computer Science Course

TL;DR

The paper addresses how integrating GenAI into an introductory CS course affects student learning and outcomes. It designs a CS1-LLM course that embeds GenAI tools like Copilot, establishes targeted design principles and learning objectives, and evaluates outcomes against international CS1 benchmarks as well as demographic groups. Key findings show exam performance largely matches historical CS1 norms, while large, open-ended projects with GenAI usage help avoid achievement gaps across demographics; students report broad usefulness of Copilot but express concerns about overreliance and uneven access during assessments. The work provides practical guidance for educators and course designers on implementing GenAI-enabled CS1, balancing foundational knowledge with AI-assisted practices and aligning assessments with industry practices.

Abstract

Generative AI (GenAI) models have broad implications for education in general, impacting the foundations of what we teach and how we assess. This is especially true in computing, where LLMs tuned for coding have demonstrated shockingly good performance on the types of assignments historically used in introductory CS (CS1) courses. As a result, CS1 courses will need to change what skills are taught and how they are assessed. Computing education researchers have begun to study student use of LLMs, but there remains much to be understood about the ways that these tools affect student outcomes. In this paper, we present the design and evaluation of a new CS1 course at a large research-intensive university that integrates the use of LLMs as a learning tool for students. We describe the design principles used to create our new CS1-LLM course, our new course objectives, and evaluation of student outcomes and perceptions throughout the course as measured by assessment scores and surveys. Our findings suggest that 1) student exam performance outcomes, including differences among demographic groups, are largely similar to historical outcomes for courses without integration of LLM tools, 2) large, open-ended projects may be particularly valuable in an LLM context, and 3) students predominantly found the LLM tools helpful, although some had concerns regarding over-reliance on the tools.
Paper Structure (45 sections, 1 equation, 4 figures, 4 tables)

This paper contains 45 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Generative AI in CS1
  4. Differences in Student Outcomes in CS1
  5. Prior Experience
  6. English as a Second Language
  7. Socioeconomic Status
  8. Race and Gender
  9. Changing CS1 Learning Objectives in Light of GenAI
  10. Course Design
  11. Design Principles for CS1-LLM
  12. Design Principle 1 -- Enable and Encourage Use of LLMs:
  13. Design Principle 2 -- Enable Students to Meet Industry Demands:
  14. Design Principle 3 -- Reduce Equity Gaps in Student Outcomes:
  15. Design Principle 4 -- Allow Creative and Personally Relevant Course Projects:
  16. ...and 30 more sections

Figures (4)

  • Figure 1: Design Principles for the course
  • Figure 2: simon2016benchmarking Benchmarking Data Results
  • Figure 3: Students perceptions of how often professionals use GenAI tools
  • Figure 4: Average exam and project grades