Evaluating 21st-Century Competencies in Postsecondary Curricula with Large Language Models: Performance Benchmarking and Reasoning-Based Prompting Strategies

TL;DR

This study addresses how to evaluate 21st-century competencies within postsecondary curricula using large language models. It systematically compares multiple curriculum document types and three competency frameworks, assembling a manually annotated dataset of 7,600 curriculum-competency alignments to benchmark zero-shot LLM performance and test a reasoning-based prompting method called Curricular CoT. The findings show that detailed instructional activity descriptions are the most informative data type, while open-weight LLMs perform reasonably on coarse tasks but lag behind humans on fine-grained reasoning; Curricular CoT yields modest improvements by structuring reasoning around pedagogical elements. The work highlights the untapped potential of institutional curriculum documents for analytics, while emphasizing the need for richer, standardized data infrastructures and benchmark datasets to enable scalable, reliable AI-driven curricular analytics in practice. Overall, the results suggest that while general-purpose LLMs have promise for high-level competency mapping, achieving human-level precision will require data-centric improvements and targeted, pedagogy-aligned modeling approaches, possibly in hybrid human-AI pipelines.

Abstract

The growing emphasis on 21st-century competencies in postsecondary education, intensified by the transformative impact of generative AI, underscores the need to evaluate how these competencies are embedded in curricula and how effectively academic programs align with evolving workforce and societal demands. Curricular Analytics, particularly recent generative AI-powered approaches, offer a promising data-driven pathway. However, analyzing 21st-century competencies requires pedagogical reasoning beyond surface-level information retrieval, and the capabilities of large language models in this context remain underexplored. In this study, we extend prior curricular analytics research by examining a broader range of curriculum documents, competency frameworks, and models. Using 7,600 manually annotated curriculum-competency alignment scores, we assess the informativeness of different curriculum sources, benchmark general-purpose LLMs for curriculum-to-competency mapping, and analyze error patterns. We further introduce a reasoning-based prompting strategy, Curricular CoT, to strengthen LLMs' pedagogical reasoning. Our results show that detailed instructional activity descriptions are the most informative type of curriculum document for competency analytics. Open-weight LLMs achieve accuracy comparable to proprietary models on coarse-grained tasks, demonstrating their scalability and cost-effectiveness for institutional use. However, no model reaches human-level precision in fine-grained pedagogical reasoning. Our proposed Curricular CoT yields modest improvements by reducing bias in instructional keyword inference and improving the detection of nuanced pedagogical evidence in long text. Together, these findings highlight the untapped potential of institutional curriculum documents and provide an empirical foundation for advancing AI-driven curricular analytics.

Figures (10)

• Figure 1: Overall study design
• Figure 2: Distribution of human annotation scores across competency frameworks
• Figure 3: Full pipeline and example of Curricular CoT
• Figure 4: Detailed templates for each prompting strategy
• Figure 5: Percentage of curriculum documents with insufficient information (NA: competency may be relevant, but available content is insufficient to determine its presence) across competency frameworks and curriculum document types