Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Course to Skill: Evaluating LLM Performance in Curricular Analytics

Zhen Xu, Xinjin Li, Yingqi Huan, Veronica Minaya, Renzhe Yu

TL;DR

This paper evaluates how well four text-alignment strategies—TF-IDF, SBERT embeddings, zero-shot prompting across multiple LLMs, and retrieval-augmented generation (RAG)—perform on skill extraction for curricular analytics. Using a stratified sample of 400 curriculum documents and a human–LLM collaborative evaluation framework, it shows that RAG consistently delivers the best precision and ranking accuracy across document types, while zero-shot prompting often underperforms traditional NLP methods, except for GPT-4o in some cases. The study provides practical guidelines on model choice and prompting for CA tasks and highlights the need for rigorous evaluation and calibration when deploying LLM-based analytics in curricular contexts. Overall, the findings underscore the potential of LLMs, especially with retrieval augmentation, to analyze sparse or heterogeneous curricula, enabling more evidence-based curriculum design and refinement.

Abstract

Curricular analytics (CA) -- systematic analysis of curricula data to inform program and course refinement -- becomes an increasingly valuable tool to help institutions align academic offerings with evolving societal and economic demands. Large language models (LLMs) are promising for handling large-scale, unstructured curriculum data, but it remains uncertain how reliably LLMs can perform CA tasks. In this paper, we systematically evaluate four text alignment strategies based on LLMs or traditional NLP methods for skill extraction, a core task in CA. Using a stratified sample of 400 curriculum documents of different types and a human-LLM collaborative evaluation framework, we find that retrieval-augmented generation (RAG) is the top-performing strategy across all types of curriculum documents, while zero-shot prompting performs worse than traditional NLP methods in most cases. Our findings highlight the promise of LLMs in analyzing brief and abstract curriculum documents, but also reveal that their performance can vary significantly depending on model selection and prompting strategies. This underscores the importance of carefully evaluating the performance of LLM-based strategies before large-scale deployment.

From Course to Skill: Evaluating LLM Performance in Curricular Analytics

TL;DR

This paper evaluates how well four text-alignment strategies—TF-IDF, SBERT embeddings, zero-shot prompting across multiple LLMs, and retrieval-augmented generation (RAG)—perform on skill extraction for curricular analytics. Using a stratified sample of 400 curriculum documents and a human–LLM collaborative evaluation framework, it shows that RAG consistently delivers the best precision and ranking accuracy across document types, while zero-shot prompting often underperforms traditional NLP methods, except for GPT-4o in some cases. The study provides practical guidelines on model choice and prompting for CA tasks and highlights the need for rigorous evaluation and calibration when deploying LLM-based analytics in curricular contexts. Overall, the findings underscore the potential of LLMs, especially with retrieval augmentation, to analyze sparse or heterogeneous curricula, enabling more evidence-based curriculum design and refinement.

Abstract

Curricular analytics (CA) -- systematic analysis of curricula data to inform program and course refinement -- becomes an increasingly valuable tool to help institutions align academic offerings with evolving societal and economic demands. Large language models (LLMs) are promising for handling large-scale, unstructured curriculum data, but it remains uncertain how reliably LLMs can perform CA tasks. In this paper, we systematically evaluate four text alignment strategies based on LLMs or traditional NLP methods for skill extraction, a core task in CA. Using a stratified sample of 400 curriculum documents of different types and a human-LLM collaborative evaluation framework, we find that retrieval-augmented generation (RAG) is the top-performing strategy across all types of curriculum documents, while zero-shot prompting performs worse than traditional NLP methods in most cases. Our findings highlight the promise of LLMs in analyzing brief and abstract curriculum documents, but also reveal that their performance can vary significantly depending on model selection and prompting strategies. This underscores the importance of carefully evaluating the performance of LLM-based strategies before large-scale deployment.
Paper Structure (15 sections, 2 figures, 2 tables)

This paper contains 15 sections, 2 figures, 2 tables.

Table of Contents

  1. Background
  2. Data and Methods
  3. Datasets
  4. Curriculum Documents.
  5. Skill Framework.
  6. Skill Extraction
  7. Token-Based (TF-IDF):
  8. Embedding-Based (BERT):
  9. Zero-Shot Prompting (ZERO-SHOT):
  10. Retrieval-Augmented Generative (RAG):
  11. Performance Evaluation
  12. Human–LLM Collaborative Evaluation Framework.
  13. Performance Metrics.
  14. Results
  15. Discussion and Conclusion

Figures (2)

  • Figure 1: Human-LLM collaborative evaluation framework
  • Figure 2: $Precision_4$ comparison across different types of curriculum documents