How undergraduate physics students use generative AI for computational modeling

TL;DR

It is found that genAI significantly impacts several aspects of students' computational modeling, such as the planning, implementing, and debugging of computational models.

Abstract

Generative artificial intelligence (genAI) is becoming increasingly prevalent and capable in physics, particularly for programming-related tasks. How, then, does genAI affect students' computational modeling? We interviewed 19 undergraduate students who had recently completed an open-ended computational assignment that encouraged the use of genAI, asking them how they used it. We then conducted a thematic analysis of these interviews using a framework for computational modeling in physics. We found that genAI significantly impacts several aspects of students' computational modeling, such as the planning, implementing, and debugging of computational models. GenAI can also help students find resources and introduce them to new computational tools. Productive use of genAI was associated with students limiting its use to small steps in the modeling process and consistently double-checking the formulas, explanations, and code it provided. We also identified challenges students faced due to an over-reliance on genAI, such as working from false model assumptions and not spending time learning the fundamentals of computational modeling, especially debugging. Finally, we discuss implications for teaching, such as the need to teach students how to use genAI productively and to urge them to plan before they code. We also highlight the continued value of low-stakes assessment and teaching assistants for teaching computational modeling, as the task remains difficult even with the introduction of genAI.

Figures (2)

• Figure 1: How students' use of generative AI impacts the different elements of computational modeling. Adapted from Figure 2 of Phillips et al. phillips_physicality_2023. In a modeling inquiry, students iteratively cycle between production and critique practices, some of which are significantly impacted by genAI, as students either commonly used genAI when engaging in these practices, or performed the practice in a substantially different way than if they did not use genAI. Students engage in these practices to reach certain objectives, and use resources and produce products during the process, some of which are affected by genAI in moderately impactful ways.
• Figure 2: The first part of the prompt a group of students sent to ChatGPT to optimize the Poisson solver provided by the course.