Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compression is Routing: Reconstruction Error as an Intrinsic Signal for Modular Language Models

Zhongpan Tang

TL;DR

The paper proposes a 'Compression is Routing' paradigm by introducing an end-to-end Transformer Autoencoder that compresses 512 tokens into 8 latent vectors (64x) and uses reconstruction loss as an intrinsic routing signal. It demonstrates a strong, domain-dependent reconstruction accuracy gap (ID TRA 99.47%, Semi-OOD 47.76%, Full-OOD 0.57%), supports clear latent-space separation, and argues for modular scheduling of expert modules without gating networks. The work highlights geometric properties of latent representations, potential for ultra-long-context handling via VRAM reduction, and benefits for continual learning through non-interfering module mounting. It presents a foundation for scalable, interpretable modular neural networks with significant inference efficiency gains, while noting limitations and directions for future research.

Abstract

Current Large Language Models (LLMs) face three major challenges: context length limitations, high inference costs, and catastrophic forgetting during continual learning. While Mixture-of-Experts (MoE) architectures mitigate some of these conflicts, their routing mechanisms typically rely on explicitly trained auxiliary classifiers. This not only increases system complexity but also often lacks interpretability when handling mixed-domain inputs. Building upon the premise that ``Compression is Intelligence,'' this paper proposes a novel architectural philosophy: Compression is Routing. We trained an 87M-parameter end-to-end Transformer Autoencoder, achieving a 64x sequence length compression (compressing 512 tokens into 8 latent vectors). Experimental results demonstrate that this compressor possesses extreme domain discriminative capability: it achieves a reconstruction accuracy of 99.47% on the in-domain (code) validation set; accuracy drops sharply to 47.76% on a semi-out-of-distribution domain (Wiki text); and further plummets to just 0.57% on a fully out-of-distribution domain (random sequences). This extreme and systematic performance discrepancy establishes the validity of reconstruction error as an Intrinsic Distribution Fingerprint. Based on this, we propose that expert modules can be automatically scheduled using reconstruction residuals directly, without the need for explicit gating networks. This mechanism offers excellent scalability. Furthermore, this architecture provides a new perspective on ``VRAM compression'' for handling ultra-long contexts. This report aims to verify the physical validity of this foundational architecture, offering a new research perspective for the next generation of scalable modular neural networks.

Compression is Routing: Reconstruction Error as an Intrinsic Signal for Modular Language Models

TL;DR

The paper proposes a 'Compression is Routing' paradigm by introducing an end-to-end Transformer Autoencoder that compresses 512 tokens into 8 latent vectors (64x) and uses reconstruction loss as an intrinsic routing signal. It demonstrates a strong, domain-dependent reconstruction accuracy gap (ID TRA 99.47%, Semi-OOD 47.76%, Full-OOD 0.57%), supports clear latent-space separation, and argues for modular scheduling of expert modules without gating networks. The work highlights geometric properties of latent representations, potential for ultra-long-context handling via VRAM reduction, and benefits for continual learning through non-interfering module mounting. It presents a foundation for scalable, interpretable modular neural networks with significant inference efficiency gains, while noting limitations and directions for future research.

Abstract

Current Large Language Models (LLMs) face three major challenges: context length limitations, high inference costs, and catastrophic forgetting during continual learning. While Mixture-of-Experts (MoE) architectures mitigate some of these conflicts, their routing mechanisms typically rely on explicitly trained auxiliary classifiers. This not only increases system complexity but also often lacks interpretability when handling mixed-domain inputs. Building upon the premise that ``Compression is Intelligence,'' this paper proposes a novel architectural philosophy: Compression is Routing. We trained an 87M-parameter end-to-end Transformer Autoencoder, achieving a 64x sequence length compression (compressing 512 tokens into 8 latent vectors). Experimental results demonstrate that this compressor possesses extreme domain discriminative capability: it achieves a reconstruction accuracy of 99.47% on the in-domain (code) validation set; accuracy drops sharply to 47.76% on a semi-out-of-distribution domain (Wiki text); and further plummets to just 0.57% on a fully out-of-distribution domain (random sequences). This extreme and systematic performance discrepancy establishes the validity of reconstruction error as an Intrinsic Distribution Fingerprint. Based on this, we propose that expert modules can be automatically scheduled using reconstruction residuals directly, without the need for explicit gating networks. This mechanism offers excellent scalability. Furthermore, this architecture provides a new perspective on ``VRAM compression'' for handling ultra-long contexts. This report aims to verify the physical validity of this foundational architecture, offering a new research perspective for the next generation of scalable modular neural networks.

Paper Structure

This paper contains 21 sections, 2 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. End-to-End Language Autoencoder Architecture
  4. Physical Isolation and Information Bottleneck
  5. Experiments and Analysis
  6. Experimental Setup
  7. Evaluation Metrics
  8. Core Finding: The Three-Step Decay of Reconstruction Accuracy
  9. Geometric Verification and Mechanism Explanation
  10. Geometric Topology of Latent Space
  11. Global Orthogonality of Manifolds
  12. Linear Separability for Routing
  13. Sparsity and Low Intrinsic Dimension
  14. Supplementary Note
  15. Discussion
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Visualization of Manifold Geometry in Latent Space. Red points represent In-Domain data (Code), and blue points represent Out-of-Domain data (Wiki). Left: t-SNE shows local clustering characteristics. Right: UMAP shows global topological isolation. The two classes of data exhibit complete linear separability in the latent space.
  • Figure 2: Intrinsic Dimension Analysis of In-Domain (Code) Latent Vectors. The curve shows the PCA cumulative explained variance ratio. The red dashed line marks the 95% variance threshold.