NITRO: LLM Inference on Intel Laptop NPUs

TL;DR

The paper tackles the challenge of running autoregressive LLMs on Intel laptop NPUs by extending the OpenVINO ecosystem with NITRO, a Python-based framework that converts modified, tensor-based Llama-like models into statically shaped OpenVINO IR chunks. It rewrites transformer components, extends KV-Caches, and stitches multiple IR chunks into an end-to-end inference pipeline capable of text generation and chat, benchmarking performance on a Meteor Lake-equipped laptop. Key findings show that while the NPU can outperform CPU for mid-size models and offers energy-efficient tradeoffs, GPUs generally maintain superior raw speed, and quantization provides limited or negative benefits on the NPU due to architectural constraints and runtime issues. The work identifies meaningful bottlenecks (rotary embeddings, attention masks outside IR) and points to future hardware upgrades (Lunar Lake) and software optimizations (speculative decoding) to enhance local NPU-based LLM deployment and energy efficiency.

Abstract

Large Language Models (LLMs) have become essential tools in natural language processing, finding large usage in chatbots such as ChatGPT and Gemini, and are a central area of research. A particular area of interest includes designing hardware specialized for these AI applications, with one such example being the neural processing unit (NPU). In 2023, Intel released the Intel Core Ultra processor with codename Meteor Lake, featuring a CPU, GPU, and NPU system-on-chip. However, official software support for the NPU through Intel's OpenVINO framework is limited to static model inference. The dynamic nature of autoregressive token generation in LLMs is therefore not supported out of the box. To address this shortcoming, we present NITRO (NPU Inference for Transformers Optimization), a Python-based framework built on top of OpenVINO to support text and chat generation on NPUs. In this paper, we discuss in detail the key modifications made to the transformer architecture to enable inference, some performance benchmarks, and future steps towards improving the package. The code repository for NITRO can be found here: https://github.com/abdelfattah-lab/nitro.

Figures (7)

• Figure 1: High level architecture of the Meteor Lake chipset.
• Figure 2: OpenVINO Ecosystem.
• Figure 3: The NITRO Framework.
• Figure 4: Diagram of the decoder layer design. In addition to the token embedding, the KV-Cache, mask, positional embeddings are inputs to the layer. The output is the next token embedding as well as the present KV-Cache, where a new slice (in yellow) is updated.
• Figure 5: Chunking structure of the model: white represents the OpenVINO IR, and gray represents the input/outputs (attention mask and positional embeddings not represented). Because the KV-Cache output is immediately fed back into the model, we can think of the cache values as an internal variable respective to each decoder layer.