OpenLearnLM Benchmark: A Unified Framework for Evaluating Knowledge, Skill, and Attitude in Educational Large Language Models

TL;DR

OpenLearnLM Benchmark addresses the need for education-focused evaluation by introducing a theory-grounded Knowledge-Skills-Attitude framework. It deploys a large, open dataset of 124K+ items with a hierarchical Skills taxonomy and deception-resilience assessment to evaluate frontier LLMs across authentic educational tasks. Key findings show that models excel in different axes (e.g., content knowledge vs. pedagogical knowledge) and that cross-domain correlations are weak, underscoring the value of multi-axis evaluation. The benchmark provides an open platform to advance educational LLM readiness, with future work spanning multilingual contexts, real classroom data, and targeted training to close identified capability gaps.

Abstract

Large Language Models are increasingly deployed as educational tools, yet existing benchmarks focus on narrow skills and lack grounding in learning sciences. We introduce OpenLearnLM Benchmark, a theory-grounded framework evaluating LLMs across three dimensions derived from educational assessment theory: Knowledge (curriculum-aligned content and pedagogical understanding), Skills (scenario-based competencies organized through a four-level center-role-scenario-subscenario hierarchy), and Attitude (alignment consistency and deception resistance). Our benchmark comprises 124K+ items spanning multiple subjects, educational roles, and difficulty levels based on Bloom's taxonomy. The Knowledge domain prioritizes authentic assessment items from established benchmarks, while the Attitude domain adapts Anthropic's Alignment Faking methodology to detect behavioral inconsistency under varying monitoring conditions. Evaluation of seven frontier models reveals distinct capability profiles: Claude-Opus-4.5 excels in practical skills despite lower content knowledge, while Grok-4.1-fast leads in knowledge but shows alignment concerns. Notably, no single model dominates all dimensions, validating the necessity of multi-axis evaluation. OpenLearnLM provides an open, comprehensive framework for advancing LLM readiness in authentic educational contexts.

Figures (6)

• Figure 1: Prior benchmarks evaluate narrow slices of educational capability. OpenLearnLM Benchmark provides comprehensive three-axis assessment covering Knowledge, Skills, and Attitude.
• Figure 2: Overview of the OpenLearnLM Benchmark framework. The benchmark evaluates educational LLMs across three axes: Knowledge (Content and Pedagogical), Skills (6 Centers with hierarchical taxonomy), and Attitude (Standard stances and Deception detection via Alignment Faking methodology).
• Figure 3: Example Skills evaluation showing rubric application for a Learner Analysis task.
• Figure 4: Sample item from the Skills domain illustrating the hierarchical metadata structure.
• Figure 5: Analysis results (Part 1): (a) Skills performance across six educational activity centers shows Counseling as the strongest (8.78) and Assessment as the most challenging (8.42). (b) Alignment consistency evaluation reveals that four models maintain consistent behavior regardless of monitoring, while Grok-4.1-fast and Gemini-3-Pro show potential alignment faking risk. (c) Radar chart displays normalized capability profiles across five dimensions.