Collaborative Edge-to-Server Inference for Vision-Language Models

TL;DR

The paper tackles high bandwidth costs in edge-to-server vision-language model inference by introducing a two-stage, entropy-aware framework. The server first infers from a downscaled global image and assesses uncertainty using min-entropy; if uncertainty is high, it utilizes attention-guided RoI localization (ViCrop) to request a detail-preserved local image and refines the result with both inputs. Key contributions include a min-entropy–based retransmission mechanism, attention-driven RoI cropping, and extensive cross-model/dataset validation showing favorable cost–accuracy tradeoffs. The approach also remains compatible with image compression, enabling practical deployment in edge scenarios with reduced communication and computation overhead.

Abstract

We propose a collaborative edge-to-server inference framework for vision-language models (VLMs) that reduces the communication cost while maintaining inference accuracy. In typical deployments, visual data captured at edge devices (clients) is transmitted to the server for VLM inference. However, resizing the original image (global image) to match the vision encoder's input resolution often discards fine-grained details, leading to accuracy degradation. To overcome this limitation, we design a two-stage framework. In the first stage, the server performs inference on the global image and identifies a region of interest (RoI) using the VLM's internal attention. The min-entropy of the output tokens is then computed as a confidence measure to determine whether retransmission is required. If the min-entropy exceeds a predefined threshold, the server requests the edge device to send a detail-preserved local image of the RoI. The server then refines its inference by jointly leveraging the global and local images. This selective retransmission strategy ensures that only essential visual content is transmitted. Experiments across multiple VLM architectures show that the proposed framework significantly reduces communication cost while maintaining inference accuracy.

Figures (12)

• Figure 1: Overview of the VLM architecture Liu2024improved.
• Figure 2: Overview of the proposed two-stage collaborative edge-to-server inference framework. The edge device first transmits a global image to the server for initial inference. If high uncertainty is detected in the output token distribution, the server requests a local image based on an attention-derived bounding box. The edge device then crops the corresponding region from the original image and transmits it to the server. The server performs refined inference using both global and local visual features.
• Figure 3: Visualization of the ViCrop method. The left image shows the original input, the middle image presents the relative attention map corresponding to the given question with the bounding box, and the right image is the detail-preserved local image.
• Figure 4: Tradeoff between communication cost and inference accuracy for the LLaVA-1.5-7B model on (a) TextVQA+OCR and (b) POPE benchmarks. The entropy metric used is min-entropy. The proposed framework achieves a more favorable communication--accuracy tradeoff compared with the high-resolution model variant (LLaVA-1.5-HD) and the random retransmission baseline.
• Figure 5: Tradeoff between computation cost and inference accuracy for the LLaVA-1.5-7B model on (a) TextVQA+OCR and (b) POPE benchmarks. The entropy metric used is min-entropy. The proposed framework achieves a more favorable computation--accuracy tradeoff compared with the high-resolution model variant (LLaVA-1.5-HD) and the random retransmission baseline.