nncase: An End-to-End Compiler for Efficient LLM Deployment on Heterogeneous Storage Architectures

TL;DR

The paper tackles the challenge of deploying extremely large language models on heterogeneous memory architectures by proposing nncase, an end-to-end compiler that unifies optimization across targets via a NUMA-aware abstraction and an e-graph-based equality-saturation engine. It introduces three core modules—Auto Vectorize, Auto Distribution, and Auto Schedule—supported by a buffer-aware Codegen and a specialized Nncase Tensor Template Library, all guided by a Roofline-based cost model to balance memory, latency, and communication. The approach yields a near hand-optimized performance on CPUs for Qwen3 models and outperforms several mainstream frameworks, demonstrating the viability of automated, architecture-aware compilation for high-performance LLM deployment. This work advances portable, high-performance LLM inference across heterogeneous storage architectures and lays groundwork for cross-architecture automation and scalable deployment.

Abstract

The efficient deployment of large language models (LLMs) is hindered by memory architecture heterogeneity, where traditional compilers suffer from fragmented workflows and high adaptation costs. We present nncase, an open-source, end-to-end compilation framework designed to unify optimization across diverse targets. Central to nncase is an e-graph-based term rewriting engine that mitigates the phase ordering problem, enabling global exploration of computation and data movement strategies. The framework integrates three key modules: Auto Vectorize for adapting to heterogeneous computing units, Auto Distribution for searching parallel strategies with cost-aware communication optimization, and Auto Schedule for maximizing on-chip cache locality. Furthermore, a buffer-aware Codegen phase ensures efficient kernel instantiation. Evaluations show that nncase outperforms mainstream frameworks like MLC LLM and Intel IPEX on Qwen3 series models and achieves performance comparable to the hand-optimized llama.cpp on CPUs, demonstrating the viability of automated compilation for high-performance LLM deployment. The source code is available at https://github.com/kendryte/nncase.

Figures (10)

• Figure 1: Overview of nncase.
• Figure 2: A comparison of traditional term rewriting and e-graph based rewriting.
• Figure 3: e-graph based auto vectorize for the Attention-like subgraph.
• Figure 4: Illustration of SBP abstraction and device placement.
• Figure 5: Pseudo-code for the distributed strategy search space construction.