Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Native LLM and MLLM Inference at Scale on Apple Silicon

Wayner Barrios

TL;DR

This work tackles the challenge of efficient, on-device multimodal LLM inference on Apple Silicon by introducing vllm-mlx, a native MLX-based framework that exploits unified memory for zero-copy execution. It delivers continuous batching, an OpenAI-compatible API, and a content-based prefix cache for vision inputs, enabling substantial throughput improvements over state-of-the-art Apple Silicon solutions and dramatic reductions in multimodal latency. Key contributions include a comprehensive benchmarking across 10+ models, a novel content hashing scheme for image caching, and memory-aware cache management that yields up to $28x$ speedups on repeated image queries and up to $24.7x$ on video frames. The framework is released as open source to enable privacy-preserving, low-latency on-device AI agents on consumer Apple hardware, with future work targeting audio modalities, distributed inference, and energy efficiency.

Abstract

The growing adoption of Apple Silicon for machine learning development has created demand for efficient inference solutions that leverage its unique unified memory architecture. However, existing tools either lack native optimization (PyTorch MPS) or focus solely on text models (llama.cpp), leaving multimodal workloads underserved. We present vllm-mlx, a framework for efficient LLM and MLLM inference on Apple Silicon built natively on MLX. For text models, we achieve 21% to 87% higher throughput than llama.cpp across models ranging from Qwen3-0.6B to Nemotron-30B, while providing continuous batching that scales to 4.3x aggregate throughput at 16 concurrent requests. For multimodal models, we introduce content-based prefix caching that eliminates redundant vision encoding by identifying identical images through content hashing, regardless of input format. Our evaluation on Apple M4 Max demonstrates throughput of up to 525 tokens per second on text models and 28x speedup on repeated image queries, reducing multimodal latency from 21.7 seconds to under 1 second. Video analysis with up to 64 frames achieves 24.7x cache speedup. We release our implementation as open source to support efficient inference on consumer Apple hardware.

Native LLM and MLLM Inference at Scale on Apple Silicon

TL;DR

This work tackles the challenge of efficient, on-device multimodal LLM inference on Apple Silicon by introducing vllm-mlx, a native MLX-based framework that exploits unified memory for zero-copy execution. It delivers continuous batching, an OpenAI-compatible API, and a content-based prefix cache for vision inputs, enabling substantial throughput improvements over state-of-the-art Apple Silicon solutions and dramatic reductions in multimodal latency. Key contributions include a comprehensive benchmarking across 10+ models, a novel content hashing scheme for image caching, and memory-aware cache management that yields up to speedups on repeated image queries and up to on video frames. The framework is released as open source to enable privacy-preserving, low-latency on-device AI agents on consumer Apple hardware, with future work targeting audio modalities, distributed inference, and energy efficiency.

Abstract

The growing adoption of Apple Silicon for machine learning development has created demand for efficient inference solutions that leverage its unique unified memory architecture. However, existing tools either lack native optimization (PyTorch MPS) or focus solely on text models (llama.cpp), leaving multimodal workloads underserved. We present vllm-mlx, a framework for efficient LLM and MLLM inference on Apple Silicon built natively on MLX. For text models, we achieve 21% to 87% higher throughput than llama.cpp across models ranging from Qwen3-0.6B to Nemotron-30B, while providing continuous batching that scales to 4.3x aggregate throughput at 16 concurrent requests. For multimodal models, we introduce content-based prefix caching that eliminates redundant vision encoding by identifying identical images through content hashing, regardless of input format. Our evaluation on Apple M4 Max demonstrates throughput of up to 525 tokens per second on text models and 28x speedup on repeated image queries, reducing multimodal latency from 21.7 seconds to under 1 second. Video analysis with up to 64 frames achieves 24.7x cache speedup. We release our implementation as open source to support efficient inference on consumer Apple hardware.
Paper Structure (41 sections, 2 figures, 7 tables, 3 algorithms)

This paper contains 41 sections, 2 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Apple Silicon and Unified Memory
  4. MLX Framework
  5. LLM Inference on Apple Silicon
  6. Serving Challenges on Apple Silicon
  7. System Design
  8. Framework Comparison
  9. Key differentiators.
  10. Text Model Inference
  11. Continuous Batching.
  12. Streaming.
  13. Text Prefix Caching.
  14. Multimodal Inference with Prefix Caching
  15. Content-Based Hashing.
  16. ...and 26 more sections

Figures (2)

  • Figure 1: Framework capability comparison. the proposed framework (green) provides comprehensive coverage: high throughput matching mlx-lm, continuous batching like vLLM-metal, OpenAI-compatible API, plus unique multimodal support with vision caching.
  • Figure 2: Concurrency scaling on vllm-mlx. (a) Aggregate throughput scales efficiently: Qwen3-0.6B achieves 3.7x higher throughput at 16 concurrent requests. (b) Request throughput (requests/sec) increases with concurrency, showing efficient batching. Qwen3-0.6B handles 25+ requests/sec at 16 concurrent connections.