Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Practical Tensor-Network Compression Pipeline for Production-Scale Large Language Models

Sergii Kozyrev, Davyd Maiboroda

TL;DR

The paper tackles production-scale deployment bottlenecks in large language models, focusing on reducing VRAM footprint and inference latency. It presents Minima, an end-to-end compression pipeline that uses a CNN-based sensitivity predictor to guide a mixed tensor-network decomposition (Tucker, TT, TR) applied patchwise, followed by a short heal fine-tune, kernel optimization, and speculative decoding. On Qwen3-32B with an $8K$ context window, Minima reduces peak VRAM from $64$ GiB to $40$ GiB and raises throughput from around $40$ to $75$ tokens per second, with modest perplexity increases (~$3\%$) and minimal accuracy degradation. The approach is practical, does not require full retraining, and can be layered with quantization; it also points toward future shared-tensor backbones and hardware-aware TN implementations for production LLMs.

Abstract

Large language models are limited in deployment by GPU memory and inference latency. We present Minima, a production compression pipeline that learns where and how to structurally compress a Transformer and turns that compression into real serving gains. Minima trains a lightweight convolutional predictor to estimate layer- and patch-level sensitivity, applies a mixture of Tucker, tensor-train, and tensor-ring decompositions to low-sensitivity regions, performs a short healing fine-tune, and executes the resulting operators with custom Triton and CUDA kernels. The reduced memory footprint enables speculative decoding with a small draft model and a larger verifier. On Qwen3-32B at an 8k-token context window, Minima reduces peak VRAM from 64 GiB to 40 GiB. For a single active request, throughput increases from 40 tokens per second (baseline) to 50 tokens per second (Minima) and 75 tokens per second (Minima with speculative decoding). Under 50 parallel requests, throughput is 34, 44, and 53 tokens per second respectively, showing that Minima remains effective under high concurrency even when speculative decoding gains compress. We position Minima relative to recent tensor-network, low-rank plus quantization, and cross-layer sharing methods, and argue that it is a practical step toward more aggressive structural compression via shared tensor backbones with tiny per-layer adapters.

A Practical Tensor-Network Compression Pipeline for Production-Scale Large Language Models

TL;DR

The paper tackles production-scale deployment bottlenecks in large language models, focusing on reducing VRAM footprint and inference latency. It presents Minima, an end-to-end compression pipeline that uses a CNN-based sensitivity predictor to guide a mixed tensor-network decomposition (Tucker, TT, TR) applied patchwise, followed by a short heal fine-tune, kernel optimization, and speculative decoding. On Qwen3-32B with an context window, Minima reduces peak VRAM from GiB to GiB and raises throughput from around to tokens per second, with modest perplexity increases (~) and minimal accuracy degradation. The approach is practical, does not require full retraining, and can be layered with quantization; it also points toward future shared-tensor backbones and hardware-aware TN implementations for production LLMs.

Abstract

Large language models are limited in deployment by GPU memory and inference latency. We present Minima, a production compression pipeline that learns where and how to structurally compress a Transformer and turns that compression into real serving gains. Minima trains a lightweight convolutional predictor to estimate layer- and patch-level sensitivity, applies a mixture of Tucker, tensor-train, and tensor-ring decompositions to low-sensitivity regions, performs a short healing fine-tune, and executes the resulting operators with custom Triton and CUDA kernels. The reduced memory footprint enables speculative decoding with a small draft model and a larger verifier. On Qwen3-32B at an 8k-token context window, Minima reduces peak VRAM from 64 GiB to 40 GiB. For a single active request, throughput increases from 40 tokens per second (baseline) to 50 tokens per second (Minima) and 75 tokens per second (Minima with speculative decoding). Under 50 parallel requests, throughput is 34, 44, and 53 tokens per second respectively, showing that Minima remains effective under high concurrency even when speculative decoding gains compress. We position Minima relative to recent tensor-network, low-rank plus quantization, and cross-layer sharing methods, and argue that it is a practical step toward more aggressive structural compression via shared tensor backbones with tiny per-layer adapters.
Paper Structure (29 sections, 5 figures, 3 tables)

This paper contains 29 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Tensor decompositions for neural network compression
  4. Tensor networks in Transformers and LLMs
  5. Low-rank, quantization, and adaptive rank selection
  6. Cross-layer parameter sharing and dictionary learning
  7. The Minima Compression Pipeline
  8. Problem setting and design goals
  9. Stage 1: Analyze --- CNN-based sensitivity prediction
  10. Stage 2: Compress --- Mixed Tucker, TT, and TR decompositions
  11. Stage 3: Heal --- Fast accuracy recovery
  12. Stage 4: Optimize kernels --- Triton/CUDA implementation
  13. Stage 5: Speculative decoding
  14. Experimental Setup
  15. Models and datasets
  16. ...and 14 more sections

Figures (5)

  • Figure 1: The Minima compression pipeline: analyze, compress, heal, optimize kernels, and speculative decode.
  • Figure 2: CNN-predicted sensitivity map used to guide compression choices.
  • Figure 3: VRAM and parameter reductions from the Minima compression pipeline.
  • Figure 4: Throughput at 8K context under single-request and 50-way concurrent load for the baseline, Minima, and Minima + Spec models.
  • Figure 5: Ablation study showing the incremental impact of each pipeline component.