Evaluating NLP Embedding Models for Handling Science-Specific Symbolic Expressions in Student Texts

TL;DR

This paper investigates how contemporary NLP embeddings handle physics-specific symbolic expressions embedded in authentic student texts, a challenge often overlooked in educational data mining. It compares German and multilingual embeddings using two approaches: a similarity-based analysis of symbolic expressions against literal translations and related concepts, and an ML-pipeline evaluation where a fixed classifier (SVM) is trained across varying embeddings on physics concept maps. The results show substantial differences among models, with GPT-text-embedding-3-large typically outperforming others in both analyses, though the gains are moderate and context may influence outcomes. The study highlights practical implications for educational applications, including model selection, cost, privacy, and dependencies on external providers, and provides a framework for benchmarking future science-language embedding models that incorporate symbolic expressions. It also points to a need for open-source, math-aware embeddings and explicit mathematical language processing to reduce biases and improve scalable educational analytics.

Abstract

In recent years, natural language processing (NLP) has become integral to educational data mining, particularly in the analysis of student-generated language products. For research and assessment purposes, so-called embedding models are typically employed to generate numeric representations of text that capture its semantic content for use in subsequent quantitative analyses. Yet when it comes to science-related language, symbolic expressions such as equations and formulas introduce challenges that current embedding models struggle to address. Existing research studies and practical applications often either overlook these challenges or remove symbolic expressions altogether, potentially leading to biased research findings and diminished performance of practical applications. This study therefore explores how contemporary embedding models differ in their capability to process and interpret science-related symbolic expressions. To this end, various embedding models are evaluated using physics-specific symbolic expressions drawn from authentic student responses, with performance assessed via two approaches: 1) similarity-based analyses and 2) integration into a machine learning pipeline. Our findings reveal significant differences in model performance, with OpenAI's GPT-text-embedding-3-large outperforming all other examined models, though its advantage over other models was moderate rather than decisive. Overall, this study underscores the importance for educational data mining researchers and practitioners of carefully selecting NLP embedding models when working with science-related language products that include symbolic expressions. The code and (partial) data are available at https://doi.org/10.17605/OSF.IO/6XQVG.

Figures (5)

• Figure 1: Illustration of the expected embedding structure if a model $M$ is capable of accurately processing and interpreting symbolic expressions. Using Example 1 from Table \ref{['tab2']}, the corresponding embedding vectors $\mathbf{e}_{}^M$ are shown in a simplified two-dimensional embedding space (for illustration purposes). The plot includes embedding vectors for the specific symbolic expression $\text{SE}_i$, its related concept $\text{RC}_i$, an incorrect related concept $\text{IRC}_i$, and an off-topic statement $\text{OT}_i$. The angles $\alpha$, $\beta$, and $\gamma$ between embedding vectors reflect semantic similarity. Specifically, if the embedding model $M$ can accurately process and interpret a symbolic expression $\text{SE}_i$, the angle $\alpha$ between its embedding vector $\mathbf{e}_{\text{SE}_i}^M$ and that of a related concept $\mathbf{e}_{\text{RC}_i}^M$ should be smaller than the angle $\beta$ to the embedding of an incorrect related concept $\mathbf{e}_{\text{IRC}_i}^M$, and much smaller than the angle $\gamma$ to the embedding of an off-topic statement $\mathbf{e}^M_{\text{OT}_i}$.
• Figure 2: Boxplot visualization of cosine similarity values across embedding models. For each embedding model $M$, there are five boxplots representing the distributions of similarity values $S_{\cos}(\mathbf{e}_{\text{SE}_i}^M, \mathbf{e}_{\text{C}_i}^M )$ for expression-text pairs of the five categories $C\in\{\text{LT}, \text{ILT}, \text{RC}, \text{IRC}, \text{OT}\}$ with LT: literal translations, ILT: incorrect literal translations, RC: related concepts, IRC: incorrect related concepts, OT: off-topic.
• Figure 3: Receiver Operating Characteristic (ROC) curves illustrating the capability of different embedding models to distinguish between correct and incorrect categories of expression-text pairs (left: LT vs. ILT; right: RC vs. IRC) based solely on model-dependent cosine similarity values. Curves closer to the top-left corner indicate better classification performance. The corresponding AUC values, ranked from highest to lowest, are displayed in the legend next to each embedding model.
• Figure 4: Boxplot visualization of the distribution of cosine similarity differences $\Delta S_{\cos}^M$ across the evaluated embedding models $M$, as defined in Eq. (\ref{['eq:sim3']}) and Eq. (\ref{['eq:sim4']}). Left: correct literal translations (LT) vs. incorrect literal translations (ILT). Right: Correct related concepts (RC) vs. incorrect related concepts (IRC). Better embedding model performance in handling symbolic expressions is indicated by a greater portion of the distribution lying above the horizontal zero line.
• Figure 5: Performance of the SVM classifier, measured by Cohen's $\kappa$, for solely textual student responses (left) and responses including symbolic expressions (right), depending on the NLP embedding model used.