Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision-Language Models on the Edge for Real-Time Robotic Perception

Sarat Ahmad, Maryam Hafeez, Syed Ali Raza Zaidi

TL;DR

This paper tackles the latency and privacy challenges of deploying Vision–Language Models in real-world robotics by moving inference to ORAN/MEC edge infrastructure. It implements a WebRTC-based pipeline that streams multimodal data from a Unitree G1 robot to an edge node, evaluating LLaMA-3.2-11B-Vision-Instruct at the edge versus cloud, and also tests the compact Qwen2-VL-2B-Instruct for latency-critical tasks. Results show edge deployment preserves near-cloud accuracy while reducing end-to-end latency by about 5%, while the compact model achieves sub-second responsiveness at the cost of reduced reasoning capability, highlighting a latency–accuracy trade-off. The work demonstrates the practicality of edge-enabled multimodal robotics and points to hybrid strategies that leverage fast perceptual grounding with selective, deeper reasoning on larger models for real-time decision making.

Abstract

Vision-Language Models (VLMs) enable multimodal reasoning for robotic perception and interaction, but their deployment in real-world systems remains constrained by latency, limited onboard resources, and privacy risks of cloud offloading. Edge intelligence within 6G, particularly Open RAN and Multi-access Edge Computing (MEC), offers a pathway to address these challenges by bringing computation closer to the data source. This work investigates the deployment of VLMs on ORAN/MEC infrastructure using the Unitree G1 humanoid robot as an embodied testbed. We design a WebRTC-based pipeline that streams multimodal data to an edge node and evaluate LLaMA-3.2-11B-Vision-Instruct deployed at the edge versus in the cloud under real-time conditions. Our results show that edge deployment preserves near-cloud accuracy while reducing end-to-end latency by 5\%. We further evaluate Qwen2-VL-2B-Instruct, a compact model optimized for resource-constrained environments, which achieves sub-second responsiveness, cutting latency by more than half but at the cost of accuracy.

Vision-Language Models on the Edge for Real-Time Robotic Perception

TL;DR

This paper tackles the latency and privacy challenges of deploying Vision–Language Models in real-world robotics by moving inference to ORAN/MEC edge infrastructure. It implements a WebRTC-based pipeline that streams multimodal data from a Unitree G1 robot to an edge node, evaluating LLaMA-3.2-11B-Vision-Instruct at the edge versus cloud, and also tests the compact Qwen2-VL-2B-Instruct for latency-critical tasks. Results show edge deployment preserves near-cloud accuracy while reducing end-to-end latency by about 5%, while the compact model achieves sub-second responsiveness at the cost of reduced reasoning capability, highlighting a latency–accuracy trade-off. The work demonstrates the practicality of edge-enabled multimodal robotics and points to hybrid strategies that leverage fast perceptual grounding with selective, deeper reasoning on larger models for real-time decision making.

Abstract

Vision-Language Models (VLMs) enable multimodal reasoning for robotic perception and interaction, but their deployment in real-world systems remains constrained by latency, limited onboard resources, and privacy risks of cloud offloading. Edge intelligence within 6G, particularly Open RAN and Multi-access Edge Computing (MEC), offers a pathway to address these challenges by bringing computation closer to the data source. This work investigates the deployment of VLMs on ORAN/MEC infrastructure using the Unitree G1 humanoid robot as an embodied testbed. We design a WebRTC-based pipeline that streams multimodal data to an edge node and evaluate LLaMA-3.2-11B-Vision-Instruct deployed at the edge versus in the cloud under real-time conditions. Our results show that edge deployment preserves near-cloud accuracy while reducing end-to-end latency by 5\%. We further evaluate Qwen2-VL-2B-Instruct, a compact model optimized for resource-constrained environments, which achieves sub-second responsiveness, cutting latency by more than half but at the cost of accuracy.
Paper Structure (19 sections, 4 figures)

This paper contains 19 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Related Works
  3. System Design
  4. Unitree G1 Robot
  5. React Application
  6. VLM Deployment on Edge
  7. Model Selection and Configuration
  8. Deployment Pipeline
  9. Cloud Baseline
  10. Communication & Control Layer
  11. Signaling Server
  12. Media Streams and Data Channels
  13. Evaluation
  14. Evaluation Metrics
  15. Accuracy
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Overview of the system architecture for edge-deployed VLM in robotic perception, illustrating the interactions among key system components.
  • Figure 2: Overview of the dataset collected with the Unitree G1 robot in a laboratory environment, comprising 200 Q&A pairs distributed across five domains of human–robot interaction
  • Figure 3: Evaluation results: (a) Accuracy and latency of VLMs on the Robo2VLM benchmark; (b) Accuracy and latency on the robot-collected dataset; (c) End-to-end latency distributions for locally deployed LLaMA-3.2.
  • Figure 4: Per-category accuracy of VLMs on the robot-collected dataset, across human–robot interaction domains.