Exploring the Impact of a Transformer's Latent Space Geometry on Downstream Task Performance

TL;DR

Transformer latent-space geometry may contribute to downstream task performance beyond generic linguistic knowledge. The authors introduce geometry-inspired measures, notably quantized cell density (Point Patchiness) and eigenvalue-based spread (EEE), and relate them to GLUE performance using perturbations and latent-space sampling. They find a strong linear relationship between patchiness and GLUE, with predictive power for non-standard training setups, suggesting geometry can inform initialization to reduce pre-training needs. The work highlights nonlinearities and methodological caveats, pointing to future directions for differentiable clustering and broader latent-space characterization.

Abstract

It is generally thought that transformer-based large language models benefit from pre-training by learning generic linguistic knowledge that can be focused on a specific task during fine-tuning. However, we propose that much of the benefit from pre-training may be captured by geometric characteristics of the latent space representations, divorced from any specific linguistic knowledge. In this work we explore the relationship between GLUE benchmarking task performance and a variety of measures applied to the latent space resulting from BERT-type contextual language models. We find that there is a strong linear relationship between a measure of quantized cell density and average GLUE performance and that these measures may be predictive of otherwise surprising GLUE performance for several non-standard BERT-type models from the literature. These results may be suggestive of a strategy for decreasing pre-training requirements, wherein model initialization can be informed by the geometric characteristics of the model's latent space.

Figures (11)

• Figure 1: Relationship between weight perturbation (Equation \ref{['eqn:noise_add']}) on pre-trained BERT-small and the resulting average GLUE score.
• Figure 2: Sample Sequence unigram, bigram, and sentence length frequencies compared to the complete combined PennTreeBank and WikiText2 datasets.
• Figure 3: Point Patchiness (PP) shows a positive linear correlation with Average GLUE score. Pearson's $r = 0.9$ for perturbed BERT-small models and $r=0.64$ for RoBERTa models.
• Figure 4: Residual plots for Point Patchiness linear regression model with model architecture variable improved the overall data fit ($r^2=0.3$ for the PP model, $r^2=0.5$ for the +Architecture Model) .
• Figure 5: Point Patchiness linear regression model is predictive of some, but not all non-standard models from the literature.