Atomic Learning Objectives Labeling: A High-Resolution Approach for Physics Education

TL;DR

The paper tackles the issue of coarse-grained learning-object mappings in physics education by introducing an atomic LO framework that encodes problem-solving cognition as input–action–output triples across nine introductory chapters. It leverages Large Language Models to automate LO labeling and proposes four evaluation metrics to assess label quality, applying them to 131 questions from expert banks and the OpenStax textbook. The study demonstrates both strengths and limitations of LLM-based LO labeling, revealing insights into model reasoning and how LO design influences labeling accuracy. By offering a high-resolution mapping between questions and learning objectives, the work lays groundwork for granular learning GPS systems that can support intelligent tutoring and personalized STEM pathways.

Abstract

This paper introduces a novel approach to create a high-resolution "map" for physics learning: an "atomic" learning objectives (LOs) system designed to capture detailed cognitive processes and concepts required for problem solving in a college-level introductory physics course. Our method leverages Large Language Models (LLMs) for automated labeling of physics questions and introduces a comprehensive set of metrics to evaluate the quality of the labeling outcomes. The atomic LO system, covering nine chapters of an introductory physics course, uses a "subject-verb-object'' structure to represent specific cognitive processes. We apply this system to 131 questions from expert-curated question banks and the OpenStax University Physics textbook. Each question is labeled with 1-8 atomic LOs across three chapters. Through extensive experiments using various prompting strategies and LLMs, we compare automated LOs labeling results against human expert labeling. Our analysis reveals both the strengths and limitations of LLMs, providing insight into LLMs reasoning processes for labeling LOs and identifying areas for improvement in LOs system design. Our work contributes to the field of learning analytics by proposing a more granular approach to mapping learning objectives with questions. Our findings have significant implications for the development of intelligent tutoring systems and personalized learning pathways in STEM education, paving the way for more effective "learning GPS'' systems.

Figures (4)

• Figure 1: An illustration of the interface developed for human experts to label learning objectives.
• Figure 2: An illustration of F1 score across various number of LOs per question for human and GPT-4o labeled LOs. The top number represents F1 score and the bottom number represents question counts. For instance, the upper-left box indicates 15 questions are labeled by human experts with only 1 LO and those 15 questions achieve an average F1 score of 0.5. The prompting strategy adopted is CoT prompting with natural language LO format (best viewed in colors).
• Figure 3: Frequency distribution of LOs in the Energy chapter: human-labeled LOs on the left, GPT-4o generated LOs with CoT prompting and natural language format on the right.
• Figure 4: Accuracy of LOs labeled by GPT-4 using CoT prompting and natural language LO format. Only LOs appearing in five or more questions are included.