Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Context Window of Large Language Models via Decomposed Positional Vectors

Zican Dong, Junyi Li, Xin Men, Wayne Xin Zhao, Bingbing Wang, Zhen Tian, Weipeng Chen, Ji-Rong Wen

TL;DR

This study explores the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs, and designs two training-free context window extension methods, positional vector replacement and attention window extension.

Abstract

Transformer-based large language models (LLMs) typically have a limited context window, resulting in significant performance degradation when processing text beyond the length of the context window. Extensive studies have been proposed to extend the context window and achieve length extrapolation of LLMs, but there is still a lack of in-depth interpretation of these approaches. In this study, we explore the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs. By using a mean-based decomposition method, we disentangle positional vectors from hidden states of LLMs and analyze their formation and effect on attention. Furthermore, when texts exceed the context window, we analyze the change of positional vectors in two settings, i.e., direct extrapolation and context window extension. Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension. Experimental results show that our methods can effectively extend the context window length.

Exploring Context Window of Large Language Models via Decomposed Positional Vectors

TL;DR

This study explores the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs, and designs two training-free context window extension methods, positional vector replacement and attention window extension.

Abstract

Transformer-based large language models (LLMs) typically have a limited context window, resulting in significant performance degradation when processing text beyond the length of the context window. Extensive studies have been proposed to extend the context window and achieve length extrapolation of LLMs, but there is still a lack of in-depth interpretation of these approaches. In this study, we explore the positional information within and beyond the context window for deciphering the underlying mechanism of LLMs. By using a mean-based decomposition method, we disentangle positional vectors from hidden states of LLMs and analyze their formation and effect on attention. Furthermore, when texts exceed the context window, we analyze the change of positional vectors in two settings, i.e., direct extrapolation and context window extension. Based on our findings, we design two training-free context window extension methods, positional vector replacement and attention window extension. Experimental results show that our methods can effectively extend the context window length.
Paper Structure (47 sections, 12 equations, 19 figures, 9 tables, 2 algorithms)

This paper contains 47 sections, 12 equations, 19 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Transformer
  4. Positional Vector
  5. Empirical Analysis
  6. Experimental Settings
  7. Formation and Effect of Positional Vectors within Context Window
  8. Formation of Positional Vectors with Full Attention
  9. Positional Vector After the First Layer
  10. Positional Information Flow From Initial Tokens
  11. Formation of Positional Vectors with Window Attention
  12. Effect of Positional Vectors on Attention
  13. Effect of Positional Vectors on Attention Sinks
  14. Effect of Positional Vectors on Long-term Decay
  15. Effect of Positional Vectors beyond Context Window
  16. ...and 32 more sections

Figures (19)

  • Figure 1: PCA visualization of positional vectors from the 1-st and 7-th layers.
  • Figure 2: Comparison of distinct positional vectors and theoretical receptive field.
  • Figure 3: Logarithmic attention maps of TL-RoPE, and TL-NoPE.
  • Figure 4: Left: The average PPL across positions during direct extrapolation. Right: The maximum cosine similarity between positional vectors within and beyond context window during extrapolation.
  • Figure 5: Left: Attention map of TL-NoPE. Middle: Attention Scores between initial token and others in TL-NoPE. Right: Similarity of logits of positional vectors across positions in TL-NoPE.
  • ...and 14 more figures