Representation Collapse in Machine Translation Through the Lens of Angular Dispersion

TL;DR

This work incorporates an existing regularization method based on angular dispersion and demonstrates empirically that it not only mitigates collapse but also improves translation quality, and shows that quantized models exhibit similar collapse behavior and that the benefits of regularization are preserved even after quantization.

Abstract

Modern neural translation models based on the Transformer architecture are known for their high performance, particularly when trained on high-resource datasets. A standard next-token prediction training strategy, while widely adopted in practice, may lead to overlooked artifacts such as representation collapse. Previous works have shown that this problem is especially pronounced in the representation of the deeper Transformer layers, where it often fails to efficiently utilize the geometric space. Representation collapse is even more evident in end-to-end training of continuous-output neural machine translation, where the trivial solution would be to set all vectors to the same value. In this work, we analyze the dynamics of representation collapse at different levels of discrete and continuous NMT transformers throughout training. We incorporate an existing regularization method based on angular dispersion and demonstrate empirically that it not only mitigates collapse but also improves translation quality. Furthermore, we show that quantized models exhibit similar collapse behavior and that the benefits of regularization are preserved even after quantization.

Figures (6)

• Figure 1: Average cosine similarity, Rényi entropy and spherical variance for decoder output, decoder embeddings and encoder outputs for Transformer-big and Transformer-base models.
• Figure 2: F1 score for each vocabulary token's frequency bucket for en-de. Note that the F1 score for tokens with frequency $< 100$ for the model without regularization is 0.
• Figure 3: BLEU score for the de-en development set newstest17 of Transformer-base model for different parameters $\gamma$. Horizontal blue line indicates the BLEU score of the unregularized Transformer-base model.
• Figure 4: Comparison of the encoder outputs representation collapse metrics for Transformer-big and Transformer-bigfp16de-en.
• Figure 5: Representation collapse in target representations of the CoNMT models. CoNMT static refers to the model trained with frozen target embeddings.