QMC: Efficient SLM Edge Inference via Outlier-Aware Quantization and Emergent Memories Co-Design

TL;DR

The paper tackles the difficulty of deploying Small Language Models on edge devices by addressing memory hierarchy and hardware noise. It introduces QMC, a retraining-free, outlier-aware quantization paired with a heterogeneous memory stack (on-chip MRAM for outliers and off-chip dense ReRAM for inliers, plus LPDDR5 for caches), coordinated by a Model Weight Controller. Through a noise-aware, ReRAM-modeling quantization strategy and a system-level co-design, QMC delivers substantial improvements in memory usage, data movement, energy, and latency while preserving accuracy relative to state-of-the-art PTQ baselines. The results demonstrate that QMC can achieve up to 6–7x memory reduction, 7–12x improvements in energy and latency, and significant data-transfer reductions on current edge AI platforms, making on-device inference for SLMs practical and scalable.

Abstract

Deploying Small Language Models (SLMs) on edge platforms is critical for real-time, privacy-sensitive generative AI, yet constrained by memory, latency, and energy budgets. Quantization reduces model size and cost but suffers from device noise in emerging non-volatile memories, while conventional memory hierarchies further limit efficiency. SRAM provides fast access but has low density, DRAM must simultaneously accommodate static weights and dynamic KV caches, which creates bandwidth contention, and Flash, although dense, is primarily used for initialization and remains inactive during inference. These limitations highlight the need for hybrid memory organizations tailored to LLM inference. We propose Outlier-aware Quantization with Memory Co-design (QMC), a retraining-free quantization with a novel heterogeneous memory architecture. QMC identifies inlier and outlier weights in SLMs, storing inlier weights in compact multi-level Resistive-RAM (ReRAM) while preserving critical outliers in high-precision on-chip Magnetoresistive-RAM (MRAM), mitigating noise-induced degradation. On language modeling and reasoning benchmarks, QMC outperforms and matches state-of-the-art quantization methods using advanced algorithms and hybrid data formats, while achieving greater compression under both algorithm-only evaluation and realistic deployment settings. Specifically, compared against SoTA quantization methods on the latest edge AI platform, QMC reduces memory usage by 6.3x-7.3x, external data transfers by 7.6x, energy by 11.7x, and latency by 12.5x when compared to FP16, establishing QMC as a scalable, deployment-ready co-design for efficient on-device inference.

Figures (4)

• Figure 1: QMC Architecture: Heterogeneous memory system with on-chip MRAM for outlier weights, off-chip MLC ReRAM for inlier weights, and LPDDR5 for KV caches, coordinated by a unified Model Weight Controller.
• Figure 2: MLC ReRAM Error Analysis. Confusion matrices and read current distributions for 3-bit (left, states S0-S7) and 2-bit (right, states S0-S3) MLC ReRAM, showing state detection accuracy and current separation between levels.
• Figure 3: Relationship between outlier ratio and perplexity, and the corresponding normalized energy/latency.
• Figure 4: Quantization Impact on System Performance. Energy (left), latency (center), and memory capacity (right) for Hymba 1.5B dong2024hymba on WikiText merity2018scalable. QMC (2-bit/3-bit MLC) achieves 6.27$\times$--12.48$\times$ gains over FP16 and outperforms AWQ/GPTQ lin2024awqfrantar2022gptq. FP16 and existing PTQ baselines use LPDDR5-based systems. While QMC uses the proposed heterogeneous NVM hierarchy.