EdgeMM: Multi-Core CPU with Heterogeneous AI-Extension and Activation-aware Weight Pruning for Multimodal LLMs at Edge

TL;DR

EdgeMM tackles the bottlenecks of edge multimodal LLMs by co-designing a multi-core CPU with heterogeneous AI extensions: a systolic-array coprocessor for GEMM and a digital CIM coprocessor for GEMV. The approach is complemented by activation-aware weight pruning and token-length driven bandwidth management to cope with limited bandwidth and varying workloads. Implemented in 22nm, EdgeMM achieves up to 2.84x speedup over a RTX 3060 laptop GPU and 0.28 token/J energy efficiency, with 138 tokens/s throughput, and shows how heterogeneous cores plus pruning and bandwidth strategies can enable practical edge inference for MLLMs. The work provides concrete architectural, ISA, and scheduling mechanisms that balance compute and memory bottlenecks across encoder and decoder stages for edge deployments. Overall, EdgeMM offers a practical path toward efficient, real-time multimodal reasoning at the edge with notable performance and energy benefits over conventional mobile GPUs.

Abstract

Emerging multimodal LLMs (MLLMs) exhibit strong cross-modality perception and reasoning capabilities and hold great potential for various applications at edge. However, MLLMs typically consist of a compute-intensive modality encoder and a memory-bound LLM decoder, leading to distinct bottlenecks for hardware designs. In this work, we present a multi-core CPU solution with heterogeneous AI extensions, which are based on either the compute-centric systolic array or memory-centric digital compute-in-memory (CIM) co-processors. In addition, dynamic activation-aware weight pruning and bandwidth management are developed to enhance bandwidth efficiency and core utilization, improving overall performance. We implemented our solution using commercial 22nm technology. For representative MLLMs, our evaluations show EdgeMM can achieve 2.84x performance speedup compared to laptop 3060 GPU.

Figures (13)

• Figure 1: MLLMs and hardware architecture design choices.
• Figure 2: Workload analysis of two MLLMs: (a) latency breakdown, (b) model statistics, (c) memory accesses.
• Figure 3: Gated-MLP and the activation vector sparsity in FFN.
• Figure 4: Architecture overview of EdgeMM design.
• Figure 5: Compute-centric core with Systolic Array.