Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Autoregressivity in the Latent Space of a GP-VAE Language Model: An Empirical Ablation Study

Yves Ruffenach

TL;DR

This study empirically interrogates latent autoregression in GP-VAE language models by contrasting an autoregressive latent dynamics variant with a non-autoregressive latent ablation and a token-space autoregressive Transformer under fixed capacity. The results show that latent autoregression yields trajectories that align strongly with the Gaussian-process prior, producing smoother, more coherent long-range structure and avoiding catastrophic loops that plague non-AR configurations. Across WikiText-2 and WikiText-103, AR latents maintain high prior density, exhibit meaningful temporal correlations, and generate more stable continuations, albeit with internal perplexities higher than a Transformer-based evaluator; non-AR latents collapse on very long horizons and are poorly aligned with the GP prior. The findings imply that a significant portion of sequential structure can be carried in latent space, suggesting hybrid architectures where latent dynamics handle long-range coherence while a lightweight decoder provides parallel token generation. They also caution against using autoregressive token-based metrics to evaluate non-autoregressive latent models, given fundamental differences in factorization and evaluation biases.

Abstract

This paper provides an ablation-based analysis of latent autoregression in GP-VAE models, building upon our previous work introducing the architecture. Language models typically rely on an autoregressive factorization over tokens. In contrast, our prior work proposed shifting sequential structure to the latent space through a causal Gaussian process, while using a non-autoregressive decoder. Here, we conduct a systematic ablation study of the role played by latent autoregression. We compare (i) a full GP-VAE model with autoregressive latent dynamics, (ii) a non-autoregressive ablation in which latent variables are independent, and (iii) a standard token-level autoregressive Transformer. Our results show that, within the considered regime (medium-scale corpora and short training contexts), latent autoregression induces latent trajectories that are significantly more compatible with the Gaussian-process prior and exhibit greater long-horizon stability. In contrast, removing autoregression leads to degraded latent structure and unstable long-range behavior. These findings highlight the role of latent autoregression as an effective mechanism for organizing long-range structure, while remaining complementary to token-level autoregressive modeling. They should be interpreted as an empirical analysis of representational structure rather than as a proposal for a new architecture.

Autoregressivity in the Latent Space of a GP-VAE Language Model: An Empirical Ablation Study

TL;DR

This study empirically interrogates latent autoregression in GP-VAE language models by contrasting an autoregressive latent dynamics variant with a non-autoregressive latent ablation and a token-space autoregressive Transformer under fixed capacity. The results show that latent autoregression yields trajectories that align strongly with the Gaussian-process prior, producing smoother, more coherent long-range structure and avoiding catastrophic loops that plague non-AR configurations. Across WikiText-2 and WikiText-103, AR latents maintain high prior density, exhibit meaningful temporal correlations, and generate more stable continuations, albeit with internal perplexities higher than a Transformer-based evaluator; non-AR latents collapse on very long horizons and are poorly aligned with the GP prior. The findings imply that a significant portion of sequential structure can be carried in latent space, suggesting hybrid architectures where latent dynamics handle long-range coherence while a lightweight decoder provides parallel token generation. They also caution against using autoregressive token-based metrics to evaluate non-autoregressive latent models, given fundamental differences in factorization and evaluation biases.

Abstract

This paper provides an ablation-based analysis of latent autoregression in GP-VAE models, building upon our previous work introducing the architecture. Language models typically rely on an autoregressive factorization over tokens. In contrast, our prior work proposed shifting sequential structure to the latent space through a causal Gaussian process, while using a non-autoregressive decoder. Here, we conduct a systematic ablation study of the role played by latent autoregression. We compare (i) a full GP-VAE model with autoregressive latent dynamics, (ii) a non-autoregressive ablation in which latent variables are independent, and (iii) a standard token-level autoregressive Transformer. Our results show that, within the considered regime (medium-scale corpora and short training contexts), latent autoregression induces latent trajectories that are significantly more compatible with the Gaussian-process prior and exhibit greater long-horizon stability. In contrast, removing autoregression leads to degraded latent structure and unstable long-range behavior. These findings highlight the role of latent autoregression as an effective mechanism for organizing long-range structure, while remaining complementary to token-level autoregressive modeling. They should be interpreted as an empirical analysis of representational structure rather than as a proposal for a new architecture.
Paper Structure (78 sections, 33 equations, 3 figures, 8 tables)

This paper contains 78 sections, 33 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Operational definition and scope of collapse.
  3. Related Work
  4. Autoregressive language models
  5. Latent-variable models for text
  6. Sequential VAEs: STORN, VRNN, and SRNN.
  7. Non-autoregressive language models.
  8. Gaussian processes and latent sequential models
  9. Recap of the GP-VAE Model with Latent Autoregression
  10. Decoder factorization and parallelism.
  11. Variational objective.
  12. Experimental Protocol
  13. Independent GP-VAE Implementations
  14. (1) Pyramidal GP-VAE implementation.
  15. (2) TCN+ GP-VAE implementation.
  16. ...and 63 more sections

Figures (3)

  • Figure 1: Training dynamics on WikiText-2 (GP-VAE AR). Evolution of (a) ELBO/token showing convergence, (b) conditional training perplexity stabilizing around $\sim 3$--$4$ (log scale), (c) KL/token capped at $8$ nats with an illustrative $\beta$-annealing schedule, and (d) decomposition of likelihood terms (LL$_0$ reconstruction vs LL$_{\text{multi}}$).
  • Figure 2: Latent Space Structure (WikiText-2): AR vs non-AR. (a) Cosine autocorrelation across temporal lags reveals strong structured dependence in AR latents, while non-AR behaves as near white noise. (b) Latent step magnitude distributions show smoother trajectories under AR (smaller $\lVert z_t-z_{t-1}\rVert$). (c) Compatibility with the GP prior (critical result): average $\log p_{\mathrm{GP}}(z)$ is +2.31$\times 10^3$ for AR versus --2.66$\times 10^7$ for non-AR, producing a visually decisive separation. (d) Diagonal-prior log densities are similar, indicating the key difference is temporal structure rather than marginal scale.
  • Figure 3: Long-range generation stability and collapse. (a) Catastrophic fraction (cat_frac) versus generation length $L_{\mathrm{gen}}$: non-AR reaches 1.0 from $L_{\mathrm{gen}}=2048$ (and remains at 3072), while AR stays at 0 up to $L_{\mathrm{gen}}=3072$. The Transformer baseline is depicted as collapsed from $L_{\mathrm{gen}}=32$ (loop frequency $\approx 1$), consistent with the qualitative analysis in the text. (b) Repetition metrics (rep2/rep3) across lengths. (c) External plausibility via GPT-2 perplexity (log scale), favoring AR at all evaluated horizons.