Enhancing Math Learning in an LMS Using AI-Driven Question Recommendations

TL;DR

The study addresses personalizing math learning in an LMS by comparing three embedding-based recommendation strategies. It generates 4096-dim multimodal embeddings with Llama-3.2-11B-Vision-Instruct and evaluates cosine similarity, Self-Organizing Maps, and Gaussian Mixture Models on logged learner interactions. Results show cosine yields precise matches but lower engagement, while SOM achieves higher engagement via a balance of similarity and novelty, and GMM underperforms due to KL-based similarity misalignment. These findings inform the design of adaptive math-recommendation systems and motivate future work on hybrid or improved similarity metrics to optimize learning outcomes.

Abstract

This paper presents an AI-driven approach to enhance math learning in a modern Learning Management System (LMS) by recommending similar math questions. Deep embeddings for math questions are generated using Meta's Llama-3.2-11B-Vision-Instruct model, and three recommendation methods-cosine similarity, Self-Organizing Maps (SOM), and Gaussian Mixture Models (GMM)-are applied to identify similar questions. User interaction data, including session durations, response times, and correctness, are used to evaluate the methods. Our findings suggest that while cosine similarity produces nearly identical question matches, SOM yields higher user satisfaction whereas GMM generally underperforms, indicating that introducing variety to a certain degree may enhance engagement and thereby potential learning outcomes until variety is no longer balanced reasonably, which our data about the implementations of all three methods demonstrate.

Figures (9)

• Figure 1: Illustration of high dimensional t-SNE down sampled datapoints (question vectors) of SOM determined clusters for XYZ (top left corner), KVA (top right corner), NOG (bottom left corner) and DTK (bottom right corner).
• Figure 2: A count plot shows that the study includes 1855 quiz sessions distributed among the three algorithms.
• Figure 3: Boxplots of quiz durations indicate that the central 90% (5th to 95th percentiles) of session durations range from 0.95 to 257.73 minutes. The other 10% of session durations are considered unreliable outliers. For instance, outlier session durations of up to 16,000 minutes were found, presumably because the user forgot to reset the quiz and returned days later. Note that sessions lasting less than 5 seconds were removed as outliers, and sessions without recorded durations (e.g., unfinished sessions) are not included either.
• Figure 4: This boxplot shows the distribution of the number of questions per quiz session for each recommendation algorithm. Note: Sessions lasting less than 5 seconds were removed as outliers and sessions without durations are not included either.
• Figure 5: Shows the distribution of user ratings on a provided scale from 1 to 5 where 5 is the best rating. Note: Users were not required to provide a rating. Consequently, there are not ratings for all sessions.