MeKi: Memory-based Expert Knowledge Injection for Efficient LLM Scaling

TL;DR

MeKi introduces a memory-based scaling paradigm that decouples LLM capacity from computation by storing token-level expert knowledge in a per-layer ROM and injecting it during inference. A training-time dynamic projection on global embeddings combines with a re-parameterization step that folds computations into a static memory table, enabling zero-cost inference despite expanded capacity. Across 0.6B, 1.7B, and 4B models, MeKi matches or exceeds dense baselines while preserving on-device decoding speed, demonstrating the practicality of memory-centric scaling for edge deployments. The work provides architectural design choices, ablations, and guidance on memory sizing, memory-slot placement, and fusion strategies to maximize performance under resource constraints.

Abstract

Scaling Large Language Models (LLMs) typically relies on increasing the number of parameters or test-time computations to boost performance. However, these strategies are impractical for edge device deployment due to limited RAM and NPU resources. Despite hardware constraints, deploying performant LLM on edge devices such as smartphone remains crucial for user experience. To address this, we propose MeKi (Memory-based Expert Knowledge Injection), a novel system that scales LLM capacity via storage space rather than FLOPs. MeKi equips each Transformer layer with token-level memory experts that injects pre-stored semantic knowledge into the generation process. To bridge the gap between training capacity and inference efficiency, we employ a re-parameterization strategy to fold parameter matrices used during training into a compact static lookup table. By offloading the knowledge to ROM, MeKi decouples model capacity from computational cost, introducing zero inference latency overhead. Extensive experiments demonstrate that MeKi significantly outperforms dense LLM baselines with identical inference speed, validating the effectiveness of memory-based scaling paradigm for on-device LLMs. Project homepage is at https://github.com/ningding-o/MeKi.

Figures (8)

• Figure 1: Memory-based scaling for on-device deployment. Scaling parameters of LLM makes inference much slower on NPU, which motivates our methodology of memory-based scaling by offloading static knowledge into Read-Only Memory (ROM). 1.7B-MeKi achieves an average zero-shot benchmark score of 59.7, statistically rivaling the 4B dense model's score of 60.5, while maintaining a 2.26$\times$ advantage in decoding speed.
• Figure 2: General overview of the proposed MeKi architecture. After training, all the re-parameterizable parts are merged into one static memory table to avoid FLOPs overhead.
• Figure 3: The comparison of validation loss for different methods where MeKi outperforms both baseline and Engram.
• Figure 4: (Left) We choose four potential positions to incorporate the MeKi module within our architecture. (Right) Validation loss for different position settings.
• Figure 5: Validation loss over different feature fusion strategies.