Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bare-Metal Tensor Virtualization: Overcoming the Memory Wall in Edge-AI Inference on ARM64

Bugra Kilictas, Faruk Alpay

TL;DR

This work tackles edge-device memory bottlenecks in LLM inference by introducing Bare-Metal Tensor Virtualization on ARM64, combining zero-copy model loading via mmap, a 64-byte aligned Structure-of-Arrays memory layout, and hand-tuned NEON kernels for GEMV and RMSNorm. By operating without heavy abstractions and with a virtual tensor core abstraction, the approach yields deterministic, memory-bound performance that serves as a portable, open reference for studying the memory wall on general-purpose ARM silicon, achieving over 60 tokens/second on an M2 while remaining energy-efficient and within practical latency bounds. Theoretical analysis (Roofline) supports the emphasis on data movement, while empirical results demonstrate stable latency and competitive throughput against hardware-accelerated baselines that remain opaque. The work also outlines future directions in quantization and linguistics-aware optimizations to broaden edge deployments and decoding strategies, highlighting practical impact for on-device NLP and real-time dialogue with constrained hardware.

Abstract

The deployment of Large Language Models (LLMs) on edge devices is fundamentally constrained by the "Memory Wall" the bottleneck where data movement latency outstrips arithmetic throughput. Standard inference runtimes often incur significant overhead through high-level abstractions, dynamic dispatch, and unaligned memory access patterns. In this work, we present a novel "Virtual Tensor Core" architecture implemented in software, optimized specifically for ARM64 microarchitectures (Apple Silicon). By bypassing standard library containers in favor of direct memory mapping (mmap) and implementing hand-tuned NEON SIMD kernels, we achieve a form of "Software-Defined Direct Memory Access (DMA)." Our proposed Tensor Virtualization Layout (TVL) guarantees 100% cache line utilization for weight matrices, while our zero-copy loader eliminates initialization latency. Experimental results on a 110M parameter model demonstrate a stable throughput of >60 tokens/second on M2 hardware. While proprietary hardware accelerators (e.g., Apple AMX) can achieve higher peak throughput, our architecture provides a fully open, portable, and deterministic reference implementation for studying the memory bottleneck on general-purpose ARM silicon, meeting the 200ms psycholinguistic latency threshold without opaque dependencies.

Bare-Metal Tensor Virtualization: Overcoming the Memory Wall in Edge-AI Inference on ARM64

TL;DR

This work tackles edge-device memory bottlenecks in LLM inference by introducing Bare-Metal Tensor Virtualization on ARM64, combining zero-copy model loading via mmap, a 64-byte aligned Structure-of-Arrays memory layout, and hand-tuned NEON kernels for GEMV and RMSNorm. By operating without heavy abstractions and with a virtual tensor core abstraction, the approach yields deterministic, memory-bound performance that serves as a portable, open reference for studying the memory wall on general-purpose ARM silicon, achieving over 60 tokens/second on an M2 while remaining energy-efficient and within practical latency bounds. Theoretical analysis (Roofline) supports the emphasis on data movement, while empirical results demonstrate stable latency and competitive throughput against hardware-accelerated baselines that remain opaque. The work also outlines future directions in quantization and linguistics-aware optimizations to broaden edge deployments and decoding strategies, highlighting practical impact for on-device NLP and real-time dialogue with constrained hardware.

Abstract

The deployment of Large Language Models (LLMs) on edge devices is fundamentally constrained by the "Memory Wall" the bottleneck where data movement latency outstrips arithmetic throughput. Standard inference runtimes often incur significant overhead through high-level abstractions, dynamic dispatch, and unaligned memory access patterns. In this work, we present a novel "Virtual Tensor Core" architecture implemented in software, optimized specifically for ARM64 microarchitectures (Apple Silicon). By bypassing standard library containers in favor of direct memory mapping (mmap) and implementing hand-tuned NEON SIMD kernels, we achieve a form of "Software-Defined Direct Memory Access (DMA)." Our proposed Tensor Virtualization Layout (TVL) guarantees 100% cache line utilization for weight matrices, while our zero-copy loader eliminates initialization latency. Experimental results on a 110M parameter model demonstrate a stable throughput of >60 tokens/second on M2 hardware. While proprietary hardware accelerators (e.g., Apple AMX) can achieve higher peak throughput, our architecture provides a fully open, portable, and deterministic reference implementation for studying the memory bottleneck on general-purpose ARM silicon, meeting the 200ms psycholinguistic latency threshold without opaque dependencies.
Paper Structure (45 sections, 4 equations, 2 figures, 1 table)

This paper contains 45 sections, 4 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. The Transformer Architecture
  4. Systems Optimization for Deep Learning
  5. Theoretical Framework
  6. Roofline Model Analysis
  7. Cache Line Alignment
  8. System Architecture
  9. Tensor Virtualization via Address Mapping
  10. Virtual Register File (RunState)
  11. Implementation Challenges
  12. Architectural Heterogeneity: Weight Tying
  13. Numerical Stability in NEON Accumulation
  14. SIMD Kernel Implementation
  15. Matrix-Vector Multiplication (GEMV)
  16. ...and 30 more sections

Figures (2)

  • Figure 1: Throughput Comparison (Tokens/Second). While hardware-accelerated PyTorch utilizes the Apple AMX coprocessor for extreme throughput, our Bare-Metal implementation demonstrates a $2.5\times$ speedup over Scalar C++ using only standard NEON instructions.
  • Figure 2: Token Generation Latency Stability ($N=256$). The tight spread between P50 and P99 indicates deterministic execution.