Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Chatbot Conversations in Physics Education: Using Artificial Intelligence to Analyze Student Reasoning through Computational Grounded Theory

Atharva Dange, Ramon E. Lopez

TL;DR

CGT is applied to analyze student misconceptions using interaction data from an AI-powered chatbot deployed in a university-level Modern Physics course to underscore the potential of CGT as a scalable, theory-aligned approach for extracting insights from chatbot dialogues and guiding the development of more adaptive, AI-driven educational tools in physics instruction.

Abstract

This study applies Computational Grounded Theory (CGT) to analyze student misconceptions using interaction data from an AI-powered chatbot deployed in a university-level Modern Physics course. The chatbot - the UTA Study Buddy Bot - engaged students in peer-like problem-solving conversations throughout the semester, generating a rich dataset of over 10 million tokens. To explore patterns in student reasoning and identify recurring conceptual difficulties, we implemented a CGT pipeline that combined natural language processing, unsupervised clustering of sentence-level vector embeddings, human interpretation of emergent themes, and supervised learning to evaluate the generalizability of identified categories. Preliminary results revealed persistent misconceptions in areas such as relativistic momentum and quantum energy levels, along with distinctive trends in how students phrased their questions and expressed uncertainty. These findings underscore the potential of CGT as a scalable, theory-aligned approach for extracting insights from chatbot dialogues and guiding the development of more adaptive, AI-driven educational tools in physics instruction.

Chatbot Conversations in Physics Education: Using Artificial Intelligence to Analyze Student Reasoning through Computational Grounded Theory

TL;DR

CGT is applied to analyze student misconceptions using interaction data from an AI-powered chatbot deployed in a university-level Modern Physics course to underscore the potential of CGT as a scalable, theory-aligned approach for extracting insights from chatbot dialogues and guiding the development of more adaptive, AI-driven educational tools in physics instruction.

Abstract

This study applies Computational Grounded Theory (CGT) to analyze student misconceptions using interaction data from an AI-powered chatbot deployed in a university-level Modern Physics course. The chatbot - the UTA Study Buddy Bot - engaged students in peer-like problem-solving conversations throughout the semester, generating a rich dataset of over 10 million tokens. To explore patterns in student reasoning and identify recurring conceptual difficulties, we implemented a CGT pipeline that combined natural language processing, unsupervised clustering of sentence-level vector embeddings, human interpretation of emergent themes, and supervised learning to evaluate the generalizability of identified categories. Preliminary results revealed persistent misconceptions in areas such as relativistic momentum and quantum energy levels, along with distinctive trends in how students phrased their questions and expressed uncertainty. These findings underscore the potential of CGT as a scalable, theory-aligned approach for extracting insights from chatbot dialogues and guiding the development of more adaptive, AI-driven educational tools in physics instruction.
Paper Structure (17 sections, 11 figures, 3 tables)

This paper contains 17 sections, 11 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Computational Grounded Theory
  3. Course Context and Instructional Design
  4. Methodology
  5. BERTopic
  6. Case Example: Month of September 2024
  7. Usage Statistics
  8. Pattern Creation
  9. Topic Outputs and Interpretive Insights
  10. Semester Wide Analysis
  11. Chatbot Usage Statistics Across the Fall 2024 Semester
  12. Topic Modeling across the Fall 2024 Semester
  13. Aggregating Fine-Grained Topics into macro-themes
  14. Visualizing and Interpreting macro-themes
  15. Pattern Confirmation and Accuracy Analysis
  16. ...and 2 more sections

Figures (11)

  • Figure 1: CGT Pipeline for Clustering Student Reasoning via BERTopic
  • Figure 2: Example of a typed question within the UTA Study Buddy Bot interface
  • Figure 3: BERTopic Flowchart for Coherent Topic Clustering
  • Figure 4: Chatbot Usage and Associated Costs Throughout September
  • Figure 5: UMAP Projection of Student Interactions (September Dataset)
  • ...and 6 more figures