Modality Inflation: Energy Characterization and Optimization Opportunities for MLLM Inference

TL;DR

This paper provides the first detailed, stage-level analysis of energy consumption in MLLM inference by breaking the pipeline into vision encoding, prefill, and decoding stages, and demonstrates that stage-wise dynamic voltage and frequency scaling (DVFS) is an effective optimization, allowing energy savings with only modest performance impact.

Abstract

Multimodal large language models (MLLMs) are built on text-only LLMs by incorporating additional modalities, enabling multimodal understanding and a broader range of applications. However, these additions introduce a previously unexplored energy trade-off across modalities that remains poorly understood, as most prior work focuses on text-only models. In this paper, we examine modality inflation, a key source of inefficiency in which multimodal inputs increase inference workloads through extra encoding stages and expanded token sequences. We provide the first detailed, stage-level analysis of energy consumption in MLLM inference by breaking the pipeline into vision encoding, prefill, and decoding stages. Using four representative MLLMs evaluated on NVIDIA A100 GPU, we quantify the additional energy required for multimodal inference compared to text-only baselines, observing overheads ranging from 17% to 94% across models for identical inputs. Our results show that energy bottlenecks differ widely across model architectures, stemming either from compute-heavy vision encoders or from the downstream impact of large visual token sequences during prefill. By examining GPU power traces, we further uncover substantial GPU underutilization during multimodal execution and show that input complexity leads to markedly different energy scaling behaviors across models. Finally, we demonstrate that stage-wise dynamic voltage and frequency scaling (DVFS) is an effective optimization, allowing energy savings with only modest performance impact. Together, these findings offer practical insights and concrete guidance for designing more energy-efficient multimodal LLM serving systems.

Figures (8)

• Figure 1: The MLLM inference pipeline. Multimodal inputs introduce a vision encoder and visual tokens (v), which are concatenated with text tokens (t) and inflate downstream prefill computation compared to text-only LLMs.
• Figure 2: Analysis of workload heterogeneity in multimodal requests.
• Figure 3: Iso-token comparison of (a) energy per request and (b) latency per request between multimodal models and their text-only baselines. The total input token count is matched by equating text-only tokens with the combined text and visual tokens in the multimodal setting, and output generation is fixed to one token. Numeric labels inside the LLM bars indicate the number of tokens used for iso-token matching.
• Figure 4: Stage-wise breakdown of inference into encoding, prefill, and decoding with a fixed output length, showing (a) energy per request and (b) latency per request for each stage. Numeric labels inside the prefill bars indicate the number of visual tokens produced by each model in this setting.
• Figure 5: Normalized GPU power traces for text-only LLM and multimodal LLM inference using batch size 32 and output length 32 to drive the GPU toward saturation.