Stop Jostling: Adaptive Negative Sampling Reduces the Marginalization of Low-Resource Language Tokens by Cross-Entropy Loss

TL;DR

This work identifies marginalization of rare tokens as a key bottleneck for language models trained on low-resource languages and introduces adaptive negative sampling via logit thresholding to reduce harmful non-target updates. By thresholding logits after the classifier head, the method suppresses gradients from irrelevant tokens, allowing rare-token embeddings to learn more meaningful representations. The approach is enhanced by separating embeddings per token and is supported by analyses of gradient flow, long-tail dynamics, and cross-language translation-like clustering, with substantial gains on low-resource validation data. The technique offers a practical path to more balanced performance in multilingual language models, particularly for underrepresented languages, while highlighting trade-offs in perplexity and tail behavior that can be mitigated with temperature and sampling adjustments.

Abstract

Neural language models often struggle with low-resource languages due to the limited availability of training data, making tokens from these languages rare in the training set. This paper addresses a specific challenge during training: rare tokens are disproportionately affected by marginalization, which prevents them from learning effectively. We propose a thresholding technique that reduces the impact of this marginalization, allowing rare tokens to benefit from more meaningful alignment. Through experiments with a character-level language model, we demonstrate that this method significantly improves performance on low-resource language validation data. This work is the first to show how negative sampling can be applied to improve the representation of rare tokens by limiting the harmful influence of excessive marginalization, offering a new approach to enhancing language model performance for underrepresented languages.

Figures (8)

• Figure 1: Three embeddings optimization types. Example for $X$ = 'don't worry, be happy' by; $Y$ = Bobby McFerrin. Context alignment (blue): adjust $w_{\text{happy}}$ so that $g_{\theta}(\dots, w_{\text{happy}}, \dots)$ moves closer to $w_{\text{Bobby}}$. Target alignment (green): move $w_{\text{Bobby}}$ closer to $g_{\theta}(\dots, w_{\text{happy}}, \dots)$. Non-target marginalization (red): move non-relevant $w_{\text{Bob}}$ and $w_{\text{cat}}$ away from $g_{\theta}(\dots, w_{\text{happy}}, \dots)$. The proposed method prevents marginalization of embeddings under the threshold.
• Figure 2: Token distribution in the experiment with a simulated low-resource (2%) and high-resource (98%) languages. For the high-resource language, expected frequencies range from 150,209.5 (token " ") to 0.98 (token "$"); for the low-resource language, expected frequencies range from 3,065.5 to 0.02.
• Figure 3: Ratio of embedding gradients norms for different tokens in the low-resource language. Tokens are sorted by frequency. Rare tokens have a lower Ratio of Average. In the baseline model, all tokens have an Average of Ratio below 1, indicating that marginalization has a strong effect on these tokens. The proposed method increases the Average of Ratio by 45% and the Ratio of Average by 12% on average.
• Figure 4: Example of real distribution of $P_{\theta}(v_i)$ from the Baseline and Proposed methods. Due to thresholding, $P_{\theta}(v_i)$ for non-relevant tokens is pushed down only until they fall below the threshold. This creates the issue of an unreliable tail, where even though $P_{\theta}(x_t)$ from the Proposed remains the highest among all tokens, its value is still lower than that of $P_{\theta}(x_t)$ from the Baseline.
• Figure 5: Expected clusters mask and cosine similarity of embeddings. The mask highlights clustering patterns: NA (13 non-alphabetical characters), UC (26 uppercase letters), LC (26 lowercase letters), ID (identity diagonal, always 1), T (translation of the same letter across languages), C (capitalization of the same letter), and T+C (capitalization of the same letter across languages). The embeddings are sorted by language and then alphabetically. The baseline model tends to marginalize low-resource (LR) embeddings, pushing them in the same direction. It only learns clusters and capitalization patterns for the high-resource language. In contrast, the proposed model captures all relationships described by the mask, revealing meaningful connections between characters across languages, without any parallel corpus in training data.