Assessing Physics Students' Scientific Argumentation using Natural Language Processing

TL;DR

This study investigates how undergraduate students' scientific argumentation in physics problem-solving evolves under progressively structured scaffolding, using unsupervised NLP to analyze open-ended essays across four semesters. The authors apply TF-IDF preprocessing and Non-negative Matrix Factorization to extract 10 topics, then track how topics concentrate over time via a concentration metric, revealing a shift from surface-level to physics-principle–based reasoning as scaffolds intensify. They validate qualitative shifts with independent EMCS pre-test data, showing that later cohorts exhibit more conceptually grounded argumentation, while early cohorts display broader distributions across procedural and conceptual themes. The findings demonstrate that scalable, data-driven analyses can facilitate feedback and improvement in argumentation-focused instruction within large physics courses, with implications for broad adoption in STEM education research and practice.

Abstract

Scientific argumentation is a core science and engineering practice and a necessary 21st Century workforce skill. Due to the nature of large enrollment classes, it is difficult to individually assess students and provide feedback on their scientific argumentation. The recent developments in Natural Language Processing (NLP) and Machine Learning (ML) provide new opportunities to analyze large collections of student writing efficiently. In this study, we investigate how undergraduate students' scientific argumentation evolves across four semesters of an introductory calculus-based physics course as increasingly structured argumentation scaffolds were introduced. We investigate the use of NLP and ML, specifically topic modeling, to analyze student scientific argumentation across those semesters. We report on the emergent themes present in each semester. Our findings show a clear shift in the thematic focus of student arguments corresponding to the level of scaffolding provided. In semesters with minimal scaffolding, students' arguments emphasized procedural and surface-level features, while semesters with explicit scaffolds exhibited greater concentration around physics-principle-based themes. These results suggest that structured scaffolding supports students in constructing more conceptually grounded scientific arguments and highlights the potential of NLP and ML as scalable approaches for evaluate broad trends in students' scientific argumentation.

Figures (4)

• Figure 1: Recitation Problem for Student Argumentation
• Figure 2: Representative words and distribution of essays in NMF topics across scaffolded semesters
• Figure 3: Concentration Factors over Four Semesters
• Figure 4: EMCS Average Scores at Start of the Semester