MMEdge: Accelerating On-device Multimodal Inference via Pipelined Sensing and Encoding

TL;DR

MMEdge targets real-time, on-device multimodal inference by introducing a pipelined sensing and encoding framework that processes data in fine-grained units, enabling encoding to overlap with sensing. It couples this pipeline with an adaptive multimodal configuration optimizer and a cross-modal speculative skipping mechanism to handle dynamic data and resource variability while maintaining accuracy. Key contributions include the decomposition of encoding into unit-level pipelines with a lightweight temporal aggregation module, offline/online optimization for modality- and model-configuration selection, and a gating-based early-skipping strategy that reduces wait times for slower modalities. Extensive UAV-based real-world testing and public dataset evaluations demonstrate significant end-to-end latency reductions (e.g., up to ~75-80%) with minimal or acceptable accuracy trade-offs, highlighting MMEdge’s practical impact for latency-constrained edge applications.

Abstract

Real-time multimodal inference on resource-constrained edge devices is essential for applications such as autonomous driving, human-computer interaction, and mobile health. However, prior work often overlooks the tight coupling between sensing dynamics and model execution, as well as the complex inter-modality dependencies. In this paper, we propose MMEdge, an new on-device multi-modal inference framework based on pipelined sensing and encoding. Instead of waiting for complete sensor inputs, MMEdge decomposes the entire inference process into a sequence of fine-grained sensing and encoding units, allowing computation to proceed incrementally as data arrive. MMEdge also introduces a lightweight but effective temporal aggregation module that captures rich temporal dynamics across different pipelined units to maintain accuracy performance. Such pipelined design also opens up opportunities for fine-grained cross-modal optimization and early decision-making during inference. To further enhance system performance under resource variability and input data complexity, MMEdge incorporates an adaptive multimodal configuration optimizer that dynamically selects optimal sensing and model configurations for each modality under latency constraints, and a cross-modal speculative skipping mechanism that bypasses future units of slower modalities when early predictions reach sufficient confidence. We evaluate MMEdge using two public multimodal datasets and deploy it on a real-world unmanned aerial vehicle (UAV)-based multimodal testbed. The results show that MMEdge significantly reduces end-to-end latency while maintaining high task accuracy across various system and data dynamics.

Figures (14)

• Figure 1: An end-to-end multimodal inference system that encompasses the entire data lifecycle on devices.
• Figure 2: Comparison between two inference frameworks.
• Figure 3: Impact of different pipelined configurations. The upper-left region indicates better strategies that achieve higher accuracy with lower latency.
• Figure 4: System Overview of MMEdge. MMEdge features a new pipelined sensing and encoding framework that decomposes the entire inference task into fine-grained units for paralell execution. It also integrates an adaptive multimodal configuration module that selects sensing and model configuration for each modality adapting to varying inputs and resource dynamics; and a cross-modal speculative skipping module to selectively skip the slower modalities.
• Figure 5: Efficient temporal aggregation through alternating shift across features from neighbor units and extract difference of features to enhance temporal correlation.