CHIME: Chiplet-based Heterogeneous Near-Memory Acceleration for Edge Multimodal LLM Inference

TL;DR

CHIME tackles the memory bottlenecks of edge multimodal LLM inference by pairing two monolithic 3D memory technologies in a chiplet-based near-memory platform. A co-designed mapping framework assigns latency-critical attention to M3D DRAM and dense FFN weights to M3D RRAM, while two-cut-point dataflow and kernel fusion minimize cross-chiplet data transfers. The approach yields large speedups and energy efficiency gains over edge GPUs and single-memory baselines, and outperforms state-of-the-art PIM accelerators in throughput. The work demonstrates the practical potential of heterogenous near-memory architectures for real-time, privacy-preserving edge AI with long-context multimodal inputs. This design enables scalable, offline inference on edge devices by strategically balancing bandwidth, capacity, and endurance across memory tiers.

Abstract

The proliferation of large language models (LLMs) is accelerating the integration of multimodal assistants into edge devices, where inference is executed under stringent latency and energy constraints, often exacerbated by intermittent connectivity. These challenges become particularly acute in the context of multimodal LLMs (MLLMs), as high-dimensional visual inputs are transformed into extensive token sequences, thereby inflating the key-value (KV) cache and imposing substantial data movement overheads to the LLM backbone. To address these issues, we present CHIME, a chiplet-based heterogeneous near-memory acceleration for edge MLLMs inference. CHIME leverages the complementary strengths of integrated monolithic 3D (M3D) DRAM and RRAM chiplets: DRAM supplies low-latency bandwidth for attention, while RRAM offers dense, non-volatile storage for weights. This heterogeneous hardware is orchestrated by a co-designed mapping framework that executes fused kernels near data, minimizing cross-chiplet traffic to maximize effective bandwidth. On FastVLM (0.6B/1.7B) and MobileVLM (1.7B/3B), CHIME achieves up to 54x speedup and up to 246x better energy efficiency per inference as compared to the edge GPU NVIDIA Jetson Orin NX. It sustains 116.5-266.5 token/J compared to Jetson's 0.7-1.1 token/J. Furthermore, it delivers up to 69.2x higher throughput than the state-of-the-art PIM accelerator FACIL. Compared to the M3D DRAM-only design, CHIME's heterogeneous memory further improves energy efficiency by 7% and performance by 2.4x.

Figures (9)

• Figure 1: MLLM software architecture overview (a) MLLM with three key features (b) Execution time breakdown of MLLMs under different connectors (c) Execution time breakdown of the GPT-2 backbone on GPU han2025sal
• Figure 2: CHIME system overview (a) 2.5D UCIe package integrating M3D DRAM and M3D RRAM on logic dies via interposer (b) Device-level tradeoffs of DRAM (1T1C) and RRAM (1T1R)
• Figure 3: M3D DRAM hardware design (a) M3D DRAM stack with NMP on logic die, organized as channels, banks, and PUs (b) Bank organization with MATs (c) Vertical M3D stacking of 1T1C subarrays
• Figure 4: M3D RRAM hardware design (a) 3D RRAM stack with NMP on logic die, organized as channels, controllers, and PUs (b) RRAM macro (c) Tile organization with local H-trees (d) Die-level layout with controllers, channels, and tiles (e) Memory subsystem with channel and tile I/O
• Figure 5: MLLM dataflow and CHIME mapping (a) MLLMs abstraction with vision encoder, connector, and transformer-based LLM (b) Mapping framework pipeline with workload-aware data layout, KV cache tiered scheduling, and kernel locality-aware fusion (c) Mapping implementation