Continuity and Isolation Lead to Doubts or Dilemmas in Large Language Models

TL;DR

This work analyzes fundamental theoretical limits of decoder-only Transformers with compact positional encoding by identifying two core phenomena: isolation, which prevents learning two nearby infinite sequences, and continuity, which fosters stability of outputs under small prompt changes. The authors prove these properties hold generally for such models and corroborate them with empirical studies across modern LLMs on zero-fundamental sequences, code-syntax verification, and periodic pattern generation. The findings imply that even simple pattern learning is inherently constrained: no single model can learn all periodic sequences or multiple nearby sequences, revealing intrinsic doubts or dilemmas in the learnability landscape. The work highlights the practical impact on prompt design, model capabilities, and the limits of current architectures, while noting that certain architectural or training adjustments (e.g., unbounded positional encodings or extended chain-of-thought) might alter these limits.

Abstract

Understanding how Transformers work and how they process information is key to the theoretical and empirical advancement of these machines. In this work, we demonstrate the existence of two phenomena in Transformers, namely isolation and continuity. Both of these phenomena hinder Transformers to learn even simple pattern sequences. Isolation expresses that any learnable sequence must be isolated from another learnable sequence, and hence some sequences cannot be learned by a single Transformer at the same time. Continuity entails that an attractor basin forms around a learned sequence, such that any sequence falling in that basin will collapse towards the learned sequence. Here, we mathematically prove these phenomena emerge in all Transformers that use compact positional encoding, and design rigorous experiments, demonstrating that the theoretical limitations we shed light on occur on the practical scale.

Figures (12)

• Figure 1: Isolation and Continuity in decoder-only Transformers.(a) Isolation: Illustration of the isolation phenomenon in the space of infinite sequences $\Sigma^\omega$. Only a few sequences (black dots) are eventually learnable by the Transformer $T$, and each is surrounded by a region where no other distinct sequence is learnable (blue dot). (b) Continuity: This figure illustrates how two similar input sequences $\alpha, \beta \in \Sigma^n$ of length $n \geq 3$ over the alphabet $\Sigma = \{\sigma_1,\sigma_2, \sigma_3\}$, which share the final token ($\alpha_n = \beta_n$) and differ in two positions ($d_H(\alpha, \beta) = 2/n \leq \delta$), are mapped by the Transformer $T$ to probability distributions with at most $\epsilon$ distance in the simplex $\Delta(\Sigma)$.
• Figure 2: Sensitivity of decoder-only language models to input perturbations at $\gamma < 0.2$ and $\gamma \geq 0.2$.
• Figure 3: Sensitivity in the SyntaxVerification task. This figure illustrates the sensitivity of five models to subtle syntactic changes in Python functions for pairs of input prompts. Each dot represents the model's probability assigned to a target token $\sigma$ under two prompts, $\alpha$ and $\beta$. Blue dots indicate sensitive cases where the model’s output changed in response to the syntactic errors, as expected. Orange dots mark non-sensitive cases where the model failed to adapt its prediction, despite the change in input. Percentages in each subplot indicate the proportion of samples where the model exhibited sensitivity.
• Figure 4: Evaluation of periodic sequence generation using Llama-2-7b-hf.
• Figure 5: Schematic of a decoder-only transformer.

Theorems & Definitions (13)

• Definition 1
• Definition 2
• Theorem 1
• Corollary 1: Next-token propagation principle, informal
• Definition 3
• Proposition 1
• Theorem 2
• Corollary 2
• proof
• Lemma 1