Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies

TL;DR

The paper proves theoretically that perplexity is invariant to the order of token factorization in autoregressive models, via the chain rule and a begin-of-sequence token, establishing a rigorous benchmark for probability consistency. It then rigorously tests this invariance with theory-aligned experiments, training forward, backward, and permuted-order GPT-2 models on a neuroscience corpus and evaluating perplexity, attention patterns, representational structure, and BrainBench performance. Empirically, forward and backward models are broadly similar but exhibit systematic deviations, while permuted-order models show pronounced inconsistencies linked to positional biases in self-attention and representation. These findings highlight how architectural and data-driven biases can undermine theoretical equivalence, informing better evaluation protocols and more trustworthy LLM behavior in practice.

Abstract

Can autoregressive large language models (LLMs) learn consistent probability distributions when trained on sequences in different token orders? We prove formally that for any well-defined probability distribution, sequence perplexity is invariant under any factorization, including forward, backward, or arbitrary permutations. This result establishes a rigorous theoretical foundation for studying how LLMs learn from data and defines principled protocols for empirical evaluation. Applying these protocols, we show that prior studies examining ordering effects suffer from critical methodological flaws. We retrain GPT-2 models across forward, backward, and arbitrary permuted orders on scientific text. We find systematic deviations from theoretical invariance across all orderings with arbitrary permutations strongly deviating from both forward and backward models, which largely (but not completely) agreed with one another. Deviations were traceable to differences in self-attention, reflecting positional and locality biases in processing. Our theoretical and empirical results provide novel avenues for understanding positional biases in LLMs and suggest methods for detecting when LLMs' probability distributions are inconsistent and therefore untrustworthy.

Figures (30)

• Figure 1: Average validation perplexity differences across across model sizes and training directions. Forward and backward text training yields similar perplexities, though forward models consistently achieve lower values (difference below zero). This gap widens slightly with model size. Permuted text training yields much higher perplexity than both forward and backward models, with similar differences to each, causing the curves to overlap. Shaded regions indicate one standard deviation over the mean across three random initializations.
• Figure 2: Attention entropy across three data orders (GPT-2 124M). Normalized attention entropy ($min=0, max=1$) is measured across layers averaged over heads and sampled text sequences for varying context sizes. Models trained on forward, backward, and permuted token orders show distinct patterns despite using the same data. Forward and backward models exhibit similar trends, with larger differences at early layers. The model trained with permuted token order displays substantially higher and more divergent entropy, particularly in early to middle layers, suggesting distinct learning dynamics driven by unnatural local and long-range dependencies. Models at initialization (Init) are shown for reference and display near-maximal entropy.
• Figure 3: Positional bias in self-attention varies with training directions and layers (GPT-2 124M). Normalized attention rank ($min=0, max=1$) is plotted as a function of token distance within the context, averaged across heads, sampled sequences, and layers. Compared to models at initialization (Init), forward (Fwd) and backward (Bwd) trained models show strong positional biases toward both nearby tokens and tokens at maximal distance, with the degree of bias varying across layers. In contrast, the model trained on permuted text (Perm) displays distinct patterns, with positional bias generally decreasing as token distance increases across most layers.
• Figure 4: Representational similarities across training directions. Forward- and backward-trained models show higher representational similarity to each other than to the model trained on permuted text. Across all comparisons, similarity declines in deeper layers, with the permuted model's representations becoming increasingly orthogonal (toward zero correlation) to the others, indicating a diverging semantic structure from models trained on forward and backward orderings.
• Figure S.1: Training perplexity differences across model sizes and training directions.