Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TernaryLM: Memory-Efficient Language Modeling via Native 1-Bit Quantization with Adaptive Layer-wise Scaling

Nisharg Nargund, Priyesh Shukla

TL;DR

The results suggest that native 1-bit training is a promising direction for efficient neural language models, and layer-wise quantization analysis showing that middle transformer layers exhibit highest compatibility with extreme quantization, informing future non-uniform precision strategies.

Abstract

Large language models (LLMs) achieve remarkable performance but demand substantial computational resources, limiting deployment on edge devices and resource-constrained environments. We present TernaryLM, a 132M parameter transformer architecture that employs native 1-bit ternary quantization {-1, 0, +1} during training, achieving significant memory reduction without sacrificing language modeling capability. Unlike post-training quantization approaches that quantize pre-trained full-precision models, TernaryLM learns quantization-aware representations from scratch using straight-through estimators and adaptive per-layer scaling factors. Our experiments demonstrate: (1) validation perplexity of 58.42 on TinyStories; (2) downstream transfer with 82.47 percent F1 on MRPC paraphrase detection; (3) 2.4x memory reduction (498MB vs 1197MB) with comparable inference latency; and (4) stable training dynamics across diverse corpora. We provide layer-wise quantization analysis showing that middle transformer layers exhibit highest compatibility with extreme quantization, informing future non-uniform precision strategies. Our results suggest that native 1-bit training is a promising direction for efficient neural language models. Code is available at https://github.com/1nisharg/TernaryLM-Memory-Efficient-Language-Modeling.

TernaryLM: Memory-Efficient Language Modeling via Native 1-Bit Quantization with Adaptive Layer-wise Scaling

TL;DR

The results suggest that native 1-bit training is a promising direction for efficient neural language models, and layer-wise quantization analysis showing that middle transformer layers exhibit highest compatibility with extreme quantization, informing future non-uniform precision strategies.

Abstract

Large language models (LLMs) achieve remarkable performance but demand substantial computational resources, limiting deployment on edge devices and resource-constrained environments. We present TernaryLM, a 132M parameter transformer architecture that employs native 1-bit ternary quantization {-1, 0, +1} during training, achieving significant memory reduction without sacrificing language modeling capability. Unlike post-training quantization approaches that quantize pre-trained full-precision models, TernaryLM learns quantization-aware representations from scratch using straight-through estimators and adaptive per-layer scaling factors. Our experiments demonstrate: (1) validation perplexity of 58.42 on TinyStories; (2) downstream transfer with 82.47 percent F1 on MRPC paraphrase detection; (3) 2.4x memory reduction (498MB vs 1197MB) with comparable inference latency; and (4) stable training dynamics across diverse corpora. We provide layer-wise quantization analysis showing that middle transformer layers exhibit highest compatibility with extreme quantization, informing future non-uniform precision strategies. Our results suggest that native 1-bit training is a promising direction for efficient neural language models. Code is available at https://github.com/1nisharg/TernaryLM-Memory-Efficient-Language-Modeling.
Paper Structure (22 sections, 6 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 22 sections, 6 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Post-Training Quantization
  4. Quantization-Aware Training
  5. 1-Bit Language Models
  6. Efficient Small Language Models
  7. Comparison with Prior Work
  8. Methodology
  9. Architecture Overview
  10. Ternary Quantization
  11. Training Protocol
  12. Experiments
  13. Pre-training Results
  14. Training Stability Across Datasets
  15. Downstream Fine-tuning
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Training loss convergence for TernaryLM across three different datasets (TinyStories, WritingPrompts and Shakespeare), demonstrating stable optimization under 1-bit ternary quantization.
  • Figure 2: Memory and latency comparison between TernaryLM and full-precision baselines.
  • Figure 3: Per-layer sparsity scores across transformer depth. Middle layers (L5-L9) exhibit highest quantization friendliness with 60-62% sparsity, while early/late layers show moderate 45-55% sparsity.
  • Figure 4: Evolution of weight distributions during training at epochs 1, 5, 10, and 15, showing convergence toward ternary clustering.