A CEFR-Inspired Classification Framework with Fuzzy C-Means To Automate Assessment of Programming Skills in Scratch

Abstract

Context: Schools, training platforms, and technology firms increasingly need to assess programming proficiency at scale with transparent, reproducible methods that support personalized learning pathways. Objective: This study introduces a pedagogical framework for Scratch project assessment, aligned with the Common European Framework of Reference (CEFR), providing universal competency levels for students and teachers alongside actionable insights for curriculum design. Method: We apply Fuzzy C-Means clustering to 2008246 Scratch projects evaluated via Dr.Scratch, implementing an ordinal criterion to map clusters to CEFR levels (A1-C2), and introducing enhanced classification metrics that identify transitional learners, enable continuous progress tracking, and quantify classification certainty to balance automated feedback with instructor review. Impact: The framework enables diagnosis of systemic curriculum gaps-notably a "B2 bottleneck" where only 13.3% of learners reside due to the cognitive load of integrating Logic Synchronization, and Data Representation--while providing certainty--based triggers for human intervention.

Figures (8)

• Figure 1: Gap statistic for $k = 2$ to $k = 12$ clusters. Monotonic increase without a clear peak and negative values for $k \leq 6$ reflect the continuous nature of programming skill distributions.
• Figure 2: Train-test performance comparison across four key metrics. Bars represent metric values for training (blue) and test (coral) sets. Difference labels show absolute and percentage changes, demonstrating minimal generalization gap.
• Figure 3: PCA projection of training set observations colored by FCM cluster assignment. Red X markers indicate CEFR-ordered centroids. Substantial overlap between adjacent levels reflects natural skill progression continuum.
• Figure 4: Enhanced classification system results on test set. (Top left) Distribution by assigned level including transition states (e.g., A1-A2, B2-C1). (Top right) Distribution by primary level (argmax equivalent). (Bottom left) Pie chart showing classification type proportions. (Bottom right) Histogram of continuous scores (1--6 scale) with CEFR level markers.
• Figure 5: Certainty analysis on test set. (Left) Distribution by categorical certainty level (Low/Medium/High). (Center) Histogram of continuous certainty values with threshold markers and mean indicator. (Right) Box plots showing certainty distributions stratified by classification type.