LLM-based Cognitive Models of Students with Misconceptions

TL;DR

The findings reveal that LLMs trained on misconception examples can efficiently learn to replicate errors, however, the training diminishes the model's ability to solve problems correctly, particularly for problem types where the misconceptions are not applicable, thus failing to satisfy second property of CSMs.

Abstract

Accurately modeling student cognition is crucial for developing effective AI-driven educational technologies. A key challenge is creating realistic student models that satisfy two essential properties: (1) accurately replicating specific misconceptions, and (2) correctly solving problems where these misconceptions are not applicable. This dual requirement reflects the complex nature of student understanding, where misconceptions coexist with correct knowledge. This paper investigates whether Large Language Models (LLMs) can be instruction-tuned to meet this dual requirement and effectively simulate student thinking in algebra. We introduce MalAlgoPy, a novel Python library that generates datasets reflecting authentic student solution patterns through a graph-based representation of algebraic problem-solving. Utilizing MalAlgoPy, we define and examine Cognitive Student Models (CSMs) - LLMs instruction tuned to faithfully emulate realistic student behavior. Our findings reveal that LLMs trained on misconception examples can efficiently learn to replicate errors. However, the training diminishes the model's ability to solve problems correctly, particularly for problem types where the misconceptions are not applicable, thus failing to satisfy second property of CSMs. We demonstrate that by carefully calibrating the ratio of correct to misconception examples in the training data - sometimes as low as 0.25 - it is possible to develop CSMs that satisfy both properties. Our insights enhance our understanding of AI-based student models and pave the way for effective adaptive learning systems.

Figures (7)

• Figure 1: Left: Problem types and corresponding algebraic expressions. Right: MalAlgoPy's graph-based model of algebraic problem-solving. This directed acyclic graph $G = (V, E_C \cup E_M)$ models cognitively faithful step-by-step simplifications as performed by students solving linear equations. $V = \{T1, \ldots, T16\}$ denotes problem types, with T1 representing the base case $Ax = B$. $E_C \subset V \times V$ represents correct transformations (grey arrows), while $E_M \subset V \times V$ represents misconception transformation (red arrows). The graph topology encodes the solution space, with all paths converging to T1. All possible correct reductions are shown as grey arrows, while only one example misconception is depicted with red arrows for clarity. For instance, T9 (ax=b(cx+d)) can correctly reduce to T7 (ex=fx+g) where $e=a$, $f=bc$, $g=bd$ (grey arrow), or incorrectly reduce via misconception M8 (distributive law applied only to the first term) where $g=d$ (red arrow). This structure enables generation of both correct solutions and "malgorithms" that reflect common student solutions.
• Figure 2: Reduction Process
• Figure 3: Learning dynamics of misconceptions in Cognitive Student Models (CSMs). Each point represents a model trained on a specific number of misconception examples. As misconception training examples increase, Misconception Accuracy (MA, solid lines) rises, demonstrating an improved ability to replicate specific misconceptions for applicable problem types. Simultaneously, Overall Correct Accuracy (OCA, dashed lines) decreases, indicating a decline in the model's ability to solve problems correctly across all types. Top: Aggregate view of all misconceptions, with MA in blue and OCA in red. Bottom: Individual misconception patterns, each in a unique color. This inverse relationship between MA and OCA highlights the challenge of simultaneously satisfying both properties of CSMs: accurately replicating misconceptions while maintaining correct problem-solving abilities for non-applicable types. This inverse relationship between MA and OCA illustrates the challenge of developing CSMs. Note that for both plots, the initial point (misconception training examples = 0) represents the base fine-tuned Llama model with OCA $\geq 90.0\%$ and MA $<5.0\%$, trained on 2000 correct examples per problem type.
• Figure 4: Left: Impact of misconception training on correct problem-solving accuracy for CSMs. Solid lines represent accuracy for problem types where the misconception is applicable (A), while dashed lines show accuracy for non-applicable (NA) types. As the number of misconception training examples increases, accuracy decreases for both A and NA problems, with A types generally more affected. This demonstrates the challenge in training CSMs that can accurately apply misconceptions while maintaining correct problem-solving abilities for non-applicable problem types. Right: Recovery of correct problem-solving abilities with introduction of correct examples. Blue lines represent Misconception Accuracy (MA), while red lines show Overall Correct Accuracy (OCA). 'S' indicates the starting point with only misconception training. The graph demonstrates how increasing the number of correct training examples alongside misconception examples enables the model to balance both MA and OCA. The rapid improvement in OCA, even with a relatively small number of correct examples, suggests an efficient method for training CSMs capable of replicating misconceptions while preserving general problem-solving skills.
• Figure 5: Learning dynamics of misconceptions in Cognitive Student Models (CSMs).Solid lines represent Misconception Accuracy (MA), while dashed lines show Overall Correct Accuracy (OCA). As the number of misconception training examples increases, MA generally rises while OCA decreases, illustrating the trade-off between acquiring misconceptions and maintaining correct problem-solving abilities. The four subplots display results for different sets of misconceptions, each represented by a unique color. This visualization demonstrates the variability in acquisition patterns across different misconception types and highlights the challenge in developing CSMs that accurately replicate specific errors while preserving general problem-solving skills.

Theorems & Definitions (1)

• Definition 1: Cognitive Student Model