Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Help or Hype? Students' Engagement and Perception of Using AI to Solve Physics Problems

Qurat-ul-Ann Mirza, N. Sanjay Rebello

TL;DR

This study investigates ChatGPT use in solving physics problems within a graded kinematics assessment. It compares unaided Problem 1 with AI-assisted Problem 2 among $N=49$ undergraduate students, with $39$ using AI. The findings show AI users achieved greater gains, with an average improvement of $+0.43$ points (vs $-1.8$ for non-users) and a $p$-value of $0.06$, suggesting a trend toward benefit; gains were associated with prompt completeness and conceptual questions, while AI missteps underscored the need for AI literacy. The work supports integrating structured AI-literacy training in STEM education to maximize benefits and mitigate risks.

Abstract

With the rise of large language models such as ChatGPT, interest has grown in understanding how these tools influence learning in STEM education, including physics. This study explores how students use ChatGPT during a physics problem-solving task embedded in a formal assessment. We analyzed patterns of AI usage and their relationship to student performance. Findings indicate that students who engaged with ChatGPT generally performed better than those who did not. Particularly, students who provided more complete and contextual prompts experienced greater benefits. Further, students who demonstrated overall positive gains collectively asked more conceptual questions than those who exhibited overall negative gains. However, the presence of incorrect AI-generated responses also underscores the importance of critically evaluating AI output. These results suggest that while AI can be a valuable aid in problem solving, its effectiveness depends significantly on how students use it, reinforcing the need to incorporate structured AI-literacy into STEM education.

Help or Hype? Students' Engagement and Perception of Using AI to Solve Physics Problems

TL;DR

This study investigates ChatGPT use in solving physics problems within a graded kinematics assessment. It compares unaided Problem 1 with AI-assisted Problem 2 among undergraduate students, with using AI. The findings show AI users achieved greater gains, with an average improvement of points (vs for non-users) and a -value of , suggesting a trend toward benefit; gains were associated with prompt completeness and conceptual questions, while AI missteps underscored the need for AI literacy. The work supports integrating structured AI-literacy training in STEM education to maximize benefits and mitigate risks.

Abstract

With the rise of large language models such as ChatGPT, interest has grown in understanding how these tools influence learning in STEM education, including physics. This study explores how students use ChatGPT during a physics problem-solving task embedded in a formal assessment. We analyzed patterns of AI usage and their relationship to student performance. Findings indicate that students who engaged with ChatGPT generally performed better than those who did not. Particularly, students who provided more complete and contextual prompts experienced greater benefits. Further, students who demonstrated overall positive gains collectively asked more conceptual questions than those who exhibited overall negative gains. However, the presence of incorrect AI-generated responses also underscores the importance of critically evaluating AI output. These results suggest that while AI can be a valuable aid in problem solving, its effectiveness depends significantly on how students use it, reinforcing the need to incorporate structured AI-literacy into STEM education.

Paper Structure

This paper contains 7 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Methods
  4. Findings & Discussion
  5. Student Performance
  6. Students' Written Prompts & Self-Reflections
  7. Conclusions, Limitations, & Future Work

Figures (2)

  • Figure 1: Comparison of students' initial prompts: Student A, who used a conceptual prompt, saw a 10-point score increase. Student B, who used a procedural prompt, saw an 8-point score decrease. Another difference between their prompts was the use of the word 'magnitude' in describing acceleration.
  • Figure 2: Students' graphs for Problems 1 and 2, alongside their ChatGPT prompt. Notable differences include the x vs. t graphs (top left in both images), the $v_y$ vs. t graph (bottom center in both images), and $a_x$ vs. t graph (top right in both images).