Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PHOTON: Hierarchical Autoregressive Modeling for Lightspeed and Memory-Efficient Language Generation

Yuma Ichikawa, Naoya Takagi, Takumi Nakagawa, Yuzi Kanazawa, Akira Sakai

TL;DR

Transformers incur high memory and latency costs for long-context generation due to KV-cache traffic and token-by-token decoding. PHOTON introduces a hierarchical autoregressive approach with a bottom-up encoder that compresses tokens into coarse latent states and a top-down, bounded-attention decoder that reconstructs fine-grained detail, enabling hierarchical prefill and chunk-local generation. The model is trained with token-prediction plus recursive reconstruction and next-context objectives to ensure cross-level consistency, and experiments show order-of-magnitude improvements in throughput-per-memory with manageable quality loss, especially in long-context and multi-query settings. This approach offers a practical, memory-efficient pathway to scaling language generation under fixed hardware budgets, with a favorable TPM-quality Pareto frontier compared to conventional and block-transformer baselines.

Abstract

Transformers operate as horizontal token-by-token scanners; at each generation step, the model attends to an ever-growing sequence of token-level states. This access pattern increases prefill latency and makes long-context decoding increasingly memory-bound, as KV-cache reads and writes dominate inference throughput rather than arithmetic computation. We propose Parallel Hierarchical Operation for Top-down Networks (PHOTON), a hierarchical autoregressive model that replaces flat scanning with vertical, multi-resolution context access. PHOTON maintains a hierarchy of latent streams: a bottom-up encoder progressively compresses tokens into low-rate contextual states, while lightweight top-down decoders reconstruct fine-grained token representations. Experimental results show that PHOTON is superior to competitive Transformer-based language models regarding the throughput-quality trade-off, offering significant advantages in long-context and multi-query tasks. This reduces decode-time KV-cache traffic, yielding up to $10^{3}\times$ higher throughput per unit memory.

PHOTON: Hierarchical Autoregressive Modeling for Lightspeed and Memory-Efficient Language Generation

TL;DR

Transformers incur high memory and latency costs for long-context generation due to KV-cache traffic and token-by-token decoding. PHOTON introduces a hierarchical autoregressive approach with a bottom-up encoder that compresses tokens into coarse latent states and a top-down, bounded-attention decoder that reconstructs fine-grained detail, enabling hierarchical prefill and chunk-local generation. The model is trained with token-prediction plus recursive reconstruction and next-context objectives to ensure cross-level consistency, and experiments show order-of-magnitude improvements in throughput-per-memory with manageable quality loss, especially in long-context and multi-query settings. This approach offers a practical, memory-efficient pathway to scaling language generation under fixed hardware budgets, with a favorable TPM-quality Pareto frontier compared to conventional and block-transformer baselines.

Abstract

Transformers operate as horizontal token-by-token scanners; at each generation step, the model attends to an ever-growing sequence of token-level states. This access pattern increases prefill latency and makes long-context decoding increasingly memory-bound, as KV-cache reads and writes dominate inference throughput rather than arithmetic computation. We propose Parallel Hierarchical Operation for Top-down Networks (PHOTON), a hierarchical autoregressive model that replaces flat scanning with vertical, multi-resolution context access. PHOTON maintains a hierarchy of latent streams: a bottom-up encoder progressively compresses tokens into low-rate contextual states, while lightweight top-down decoders reconstruct fine-grained token representations. Experimental results show that PHOTON is superior to competitive Transformer-based language models regarding the throughput-quality trade-off, offering significant advantages in long-context and multi-query tasks. This reduces decode-time KV-cache traffic, yielding up to higher throughput per unit memory.
Paper Structure (34 sections, 16 equations, 2 figures, 7 tables)

This paper contains 34 sections, 16 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. PHOTON
  3. Architecture
  4. Notation
  5. Hierarchical Encoder
  6. Context Chunker.
  7. Context Encoder.
  8. Hierarchical Decoder
  9. Context Converter.
  10. Context Decoder.
  11. Overall Model
  12. Learning Objective
  13. Next-Token Loss.
  14. Recursive Reconstruction Loss.
  15. Next-Context Loss.
  16. ...and 19 more sections

Figures (2)

  • Figure 1: Overview of PHOTON. A bottom-up encoder aggregates token-level states into progressively coarser latent representations. A top-down decoder reconstructs finer-grained representations using a context converter and bounded local autoregressive decoding, with attention strictly limited to each chunk. At inference time, PHOTON performs a hierarchical prefill to construct multi-level states, then generates by updating coarse latents and decoding tokens in parallel across chunks, reducing global KV cache growth and decode-time memory traffic.
  • Figure 2: TPM-quality trade-offs under PF and DE regimes. TPM (Throughput-per-memory) is computed as $\text{Throughput}/\text{Memory}$ (K tokens/s/GiB). (a) PF: TPM vs. Wikitext PPL (lower is better; x-axis reversed). (b) DE: TPM vs. Wikitext PPL (x-axis reversed). (c) PF: TPM vs. average zero-shot accuracy (mean of HS, SCiQ, and ARCe; higher is better). (d) DE: TPM vs. average zero-shot accuracy. The dotted curve indicates the Pareto frontier within each panel. PHOTON achieves a better TPM-quality frontier than the Vanilla and Block Transformer in all settings.