Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Separable Architecture for Continuous Token Representation in Language Models

Reza T. Batley, Sourav Saha

TL;DR

This work tackles the embedding-token bottleneck in sub-billion-parameter language models by replacing discrete token lookups with a continuous token generator, forming Leviathan, a Transformer with a Separable Neural Architecture. The generator maps token indices to a smooth latent surface via latent indexing, B-spline basis expansion, and tensor-product aggregation, enabling a parameter-efficient but expressive embedding mechanism. Empirical results on the Pile across 60–421M parameters show Leviathan attains a substantial effective capacity increase (approximately $1.5$ to $2.1$ times), achieves perplexity reductions of $6.7\%$ to $18.1\%$ in iso-body settings, and enables a depth dividend in isoparametric regimes (up to $2.11$-fold dense-equivalent size at $109$M). The findings imply decoupling vocabulary cardinality from parameter budgets can improve long-context modeling and world-models, with scalable benefits in data efficiency, while motivating further optimizations and multi-modal extensions; the approach offers a principled path toward open-vocabulary models without full retraining.

Abstract

Transformer scaling law analyses typically treat parameters as interchangeable; an abstraction that accurately predicts loss-compute relationships. Yet, in sub-billion-parameter small language models (SLMs), embedding matrices dominate the parameter budget. This work argues that this allocation is as suboptimal as it is counterintuitive. Leviathan is an architecture with a continuous embedding generator to replace the discrete lookup tables of canonical models. Evaluating on the Pile dataset under isoparametric settings, Leviathan consistently outperforms a standard, LLaMA-style architecture. By means of an empirical power-law fit, Leviathan exhibits a markedly superior effective parameter capacity. Across the regime studied, Leviathan behaves as a dense model with $1.47$ to $2.11 \times$ more parameters.

A Separable Architecture for Continuous Token Representation in Language Models

TL;DR

This work tackles the embedding-token bottleneck in sub-billion-parameter language models by replacing discrete token lookups with a continuous token generator, forming Leviathan, a Transformer with a Separable Neural Architecture. The generator maps token indices to a smooth latent surface via latent indexing, B-spline basis expansion, and tensor-product aggregation, enabling a parameter-efficient but expressive embedding mechanism. Empirical results on the Pile across 60–421M parameters show Leviathan attains a substantial effective capacity increase (approximately to times), achieves perplexity reductions of to in iso-body settings, and enables a depth dividend in isoparametric regimes (up to -fold dense-equivalent size at M). The findings imply decoupling vocabulary cardinality from parameter budgets can improve long-context modeling and world-models, with scalable benefits in data efficiency, while motivating further optimizations and multi-modal extensions; the approach offers a principled path toward open-vocabulary models without full retraining.

Abstract

Transformer scaling law analyses typically treat parameters as interchangeable; an abstraction that accurately predicts loss-compute relationships. Yet, in sub-billion-parameter small language models (SLMs), embedding matrices dominate the parameter budget. This work argues that this allocation is as suboptimal as it is counterintuitive. Leviathan is an architecture with a continuous embedding generator to replace the discrete lookup tables of canonical models. Evaluating on the Pile dataset under isoparametric settings, Leviathan consistently outperforms a standard, LLaMA-style architecture. By means of an empirical power-law fit, Leviathan exhibits a markedly superior effective parameter capacity. Across the regime studied, Leviathan behaves as a dense model with to more parameters.
Paper Structure (41 sections, 1 theorem, 13 equations, 10 figures, 2 tables)

This paper contains 41 sections, 1 theorem, 13 equations, 10 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Parametric Efficiency of Scaling
  4. Weight Tying
  5. Factorization
  6. Continuous Token Representations
  7. Architecture
  8. A Continuous Token Generator
  9. Latent compositional indexing
  10. B-spline basis expansion
  11. Tensor product aggregation
  12. Full Model Architecture
  13. Transformer Backbone Specification
  14. Experiments
  15. The Representational Prior: Iso-Body Analysis
  16. ...and 26 more sections

Key Result

Theorem 1.1

Let $\Omega = [0,1]^d$. The class of separable functions $\mathcal{A}$ is dense in $C(\Omega)$ with respect to the uniform norm $\|\cdot\|_\infty$.

Figures (10)

  • Figure 1: The generator module, replacing the standard dense input embedding matrix.
  • Figure 2: Leviathan architecture, highlighting the replacement of the input embedding matrix with a generator module.
  • Figure 3: (Left) The parameter-loss Pareto frontier under iso-body control. (Right) Percentage reduction in validation perplexity achieved by Leviathan relative to the tied Dense baseline.
  • Figure 4: Parameter allocation across components for untied Dense and Leviathan models under fixed parameter budgets. Leviathan reallocates input parameters from embeddings into deeper reasoning layers.
  • Figure 5: (Left) The parameter-loss Pareto frontier under isoparametric control. (Right) Effective Dense-equivalent size inferred from a fitted Dense parameter scaling law.
  • ...and 5 more figures

Theorems & Definitions (2)

  • Theorem 1.1: Universal Approximation of CP-Separable Neural Architectures
  • proof