MalruleLib: Large-Scale Executable Misconception Reasoning with Step Traces for Modeling Student Thinking in Mathematics

TL;DR

MalruleLib tackles the problem of modeling student thinking in mathematics by encoding documented misconceptions as executable procedures and pairing them with hundreds of diverse templates to enable cross-context evaluation. It introduces Malrule Reasoning Accuracy (MRA) to assess whether models can infer a misrule from a worked mistake and predict its next application across surface form changes, and it contrasts this with traditional correct-solution accuracy (CRA). The framework generates dual-path traces (correct and malrule-consistent reasoning) at scale, showing that while large language models excel at direct problem solving, they struggle to transfer misconception-driven procedures across templates; step traces provide consistent gains, but cross-template generalization remains a bottleneck. The work provides a scalable infrastructure and benchmark for educational AI that supports diagnosis and targeted feedback, with implications for scalable tutoring and personalized learning. By making misrules executable and templates diverse, MalruleLib enables controlled supervision and evaluation of student-modeling capabilities, highlighting the gap between solving and simulating student reasoning and informing future instruction-tuning directions.

Abstract

Student mistakes in mathematics are often systematic: a learner applies a coherent but wrong procedure and repeats it across contexts. We introduce MalruleLib, a learning-science-grounded framework that translates documented misconceptions into executable procedures, drawing on 67 learning-science and mathematics education sources, and generates step-by-step traces of malrule-consistent student work. We formalize a core student-modeling problem as Malrule Reasoning Accuracy (MRA): infer a misconception from one worked mistake and predict the student's next answer under cross-template rephrasing. Across nine language models (4B-120B), accuracy drops from 66% on direct problem solving to 40% on cross-template misconception prediction. MalruleLib encodes 101 malrules over 498 parameterized problem templates and produces paired dual-path traces for both correct reasoning and malrule-consistent student reasoning. Because malrules are executable and templates are parameterizable, MalruleLib can generate over one million instances, enabling scalable supervision and controlled evaluation. Using MalruleLib, we observe cross-template degradations of 10-21%, while providing student step traces improves prediction by 3-15%. We release MalruleLib as infrastructure for educational AI that models student procedures across contexts, enabling diagnosis and feedback that targets the underlying misconception.

Figures (2)

• Figure 1: MalruleLib framework overview. Left: Classification by NCTM Content Strand (M=Malrules, T=Templates). Right: Distribution by developmental stage showing coverage across grade levels.
• Figure 2: Performance by mathematical category. Functions is easiest (82%), Coordinate Geometry hardest (29%).