Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FoundationalASSIST: An Educational Dataset for Foundational Knowledge Tracing and Pedagogical Grounding of LLMs

Eamon Worden, Cristina Heffernan, Neil Heffernan, Shashank Sonkar

TL;DR

FoundationalASSIST introduces the first English educational dataset that provides complete question text, actual student responses, distractor selections, and Common Core alignment, enabling language-based cognitive modeling and psychometric analyses at scale ($1.7$ million interactions, $5{,}000$ students, $3{,}395$ problems, $462$ skills). By proposing six benchmark tasks spanning Knowledge Tracing and Pedagogical Grounding, the paper demonstrates that frontier LLMs currently struggle to predict student performance, anticipate exact responses, or identify problem discrimination, with only modest gains from extended reasoning. The results reveal a systematic optimistic bias in predictions, limited benefits from longer histories, and a pronounced gap in understanding item discrimination, underscoring the need for education-focused data and training. The work positions FoundationalASSIST as a foundational resource to guide future research toward reliable, scalable, and equitable AI-assisted education, including targeted collection of student-error data and improved psychometric integration.

Abstract

Can Large Language Models understand how students learn? As LLMs are deployed for adaptive testing and personalized tutoring, this question becomes urgent -- yet we cannot answer it with existing resources. Current educational datasets provide only question identifiers and binary correctness labels, rendering them opaque to LLMs that reason in natural language. We address this gap with FoundationalASSIST, the first English educational dataset providing the complete information needed for research on LLMs in education: full question text, actual student responses (not just right/wrong), records of which wrong answers students chose, and alignment to Common Core K-12 standards. These 1.7 million interactions from 5,000 students enable research directions that were previously impossible to pursue, from fine-tuning student models to analyzing misconception patterns. To demonstrate the dataset's utility, we evaluate four frontier models (GPT-OSS-120B, Llama-3.3-70B, Qwen3-Next-80B variants) on two complementary task families: Knowledge Tracing, testing whether LLMs can predict student performance on questions, and the exact answer a student will give; and \textbf{Pedagogical Grounding}, testing whether LLMs understand the properties that make assessment items effective. Our evaluation reveals significant gaps in current LLM capabilities. Every model barely achieves a trivial baseline on knowledge tracing. All models fall below random chance on item discrimination, indicating that LLMs do not understand what makes one problem more diagnostic than another. Models do show competence at judging relative difficulty (up to 68.6%), but this partial success only highlights the gaps elsewhere. These results establish that substantial advances are needed before LLMs can reliably support personalized learning at scale. We release FoundationalASSIST to support progress on these foundational challenges.

FoundationalASSIST: An Educational Dataset for Foundational Knowledge Tracing and Pedagogical Grounding of LLMs

TL;DR

FoundationalASSIST introduces the first English educational dataset that provides complete question text, actual student responses, distractor selections, and Common Core alignment, enabling language-based cognitive modeling and psychometric analyses at scale ( million interactions, students, problems, skills). By proposing six benchmark tasks spanning Knowledge Tracing and Pedagogical Grounding, the paper demonstrates that frontier LLMs currently struggle to predict student performance, anticipate exact responses, or identify problem discrimination, with only modest gains from extended reasoning. The results reveal a systematic optimistic bias in predictions, limited benefits from longer histories, and a pronounced gap in understanding item discrimination, underscoring the need for education-focused data and training. The work positions FoundationalASSIST as a foundational resource to guide future research toward reliable, scalable, and equitable AI-assisted education, including targeted collection of student-error data and improved psychometric integration.

Abstract

Can Large Language Models understand how students learn? As LLMs are deployed for adaptive testing and personalized tutoring, this question becomes urgent -- yet we cannot answer it with existing resources. Current educational datasets provide only question identifiers and binary correctness labels, rendering them opaque to LLMs that reason in natural language. We address this gap with FoundationalASSIST, the first English educational dataset providing the complete information needed for research on LLMs in education: full question text, actual student responses (not just right/wrong), records of which wrong answers students chose, and alignment to Common Core K-12 standards. These 1.7 million interactions from 5,000 students enable research directions that were previously impossible to pursue, from fine-tuning student models to analyzing misconception patterns. To demonstrate the dataset's utility, we evaluate four frontier models (GPT-OSS-120B, Llama-3.3-70B, Qwen3-Next-80B variants) on two complementary task families: Knowledge Tracing, testing whether LLMs can predict student performance on questions, and the exact answer a student will give; and \textbf{Pedagogical Grounding}, testing whether LLMs understand the properties that make assessment items effective. Our evaluation reveals significant gaps in current LLM capabilities. Every model barely achieves a trivial baseline on knowledge tracing. All models fall below random chance on item discrimination, indicating that LLMs do not understand what makes one problem more diagnostic than another. Models do show competence at judging relative difficulty (up to 68.6%), but this partial success only highlights the gaps elsewhere. These results establish that substantial advances are needed before LLMs can reliably support personalized learning at scale. We release FoundationalASSIST to support progress on these foundational challenges.
Paper Structure (99 sections, 1 equation, 5 figures, 18 tables)

This paper contains 99 sections, 1 equation, 5 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Evolution of Knowledge Tracing
  4. The Dataset Bottleneck
  5. LLMs Enter Education
  6. Psychometric Foundations
  7. The FoundationalASSIST Dataset
  8. From ASSISTments to FoundationalASSIST
  9. Scale and Scope
  10. What the Data Contains
  11. Problem Types and Their Challenges
  12. Difficulty and Baseline Performance
  13. What Makes FoundationalASSIST Different
  14. Preprocessing
  15. Comparison with Prior Datasets
  16. ...and 84 more sections

Figures (5)

  • Figure 1: Student learning histories. (a) Students complete 211--421 interactions each, averaging 344. (b) Full histories approach 32K tokens, challenging LLM context limits.
  • Figure 2: Content distribution. (a) Fill-in-the-blank problems dominate (70% of interactions), posing a harder prediction challenge than multiple-choice. (b) Skills span all Common Core math domains from grades 3--8.
  • Figure 3: KT performance vs context length (50--400 prior interactions). Left: FKT AUC-ROC remains near random baseline (0.5) regardless of history size. Right: Cognitive modeling accuracy shows no consistent improvement with longer context.
  • Figure 4: Distribution of IRT parameters across 2,548 problems. Difficulty ranges from -1.35 (easy) to 0.91 (hard); discrimination ranges from 0.01 (uninformative) to 0.91 (highly diagnostic).
  • Figure 5: Distractor selection frequencies across 236 multiple-choice problems. Some distractors capture common misconceptions (high selection), while others are rarely chosen (low effectiveness).