Modeling Programming Skills with Source Code Embeddings for Context-aware Exercise Recommendation

TL;DR

The paper addresses personalized programming assignment in large introductory courses by modeling students' knowledge through source code embeddings. It introduces a context-aware recommender that derives an eight-skill profile from recent code submissions using a multilingual embedding model and trains eight MLP classifiers to predict each skill. Problems are then ranked by similarity between the predicted skill vector and problem-vectors, with problems represented by instructor-provided reference solutions. Evaluations on 12,912 submissions from 253 students across 112 problems show that skill-based recommendations outperform correctness- and time-based baselines, with Jina embeddings providing robust predictive performance across languages and tasks, indicating practical benefits for personalized education.

Abstract

In this paper, we propose a context-aware recommender system that models students' programming skills using embeddings of the source code they submit throughout a course. These embeddings predict students' skills across multiple programming topics, producing profiles that are matched to the skills required by unseen homework problems. To generate recommendations, we compute the cosine similarity between student profiles and problem skill vectors, ranking exercises according to their alignment with each student's current abilities. We evaluated our approach using real data from students and exercises in an introductory programming course at our university. First, we assessed the effectiveness of our source code embeddings for predicting skills, comparing them with token-based and graph-based alternatives. Results showed that Jina embeddings outperformed TF-IDF, CodeBERT-cpp, and GraphCodeBERT across most skills. Additionally, we evaluated the system's ability to recommend exercises aligned with weekly course content by analyzing student submissions collected over seven course offerings. Our approach consistently produced more suitable recommendations than baselines based on correctness or solution time, indicating that predicted programming skills provide a stronger signal for problem recommendation.

Figures (1)

• Figure 1: Average percentage of suitable exercises (the higher, the better) of different ranking metrics (Skills, Solution Time, and Solution Correctness) with varying strategies for summarizing the submissions into a single embedding vector.