Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VLAgents: A Policy Server for Efficient VLA Inference

Tobias Jülg, Khaled Gamal, Nisarga Nilavadi, Pierre Krack, Seongjin Bien, Michael Krawez, Florian Walter, Wolfram Burgard

TL;DR

The paper tackles fragmentation and latency in Vision-Language-Action (VLA) robotics by introducing VLAgents, a model-agnostic policy server with a Gymnasium-like interface that bridges VLAs to both simulated and real robots. Its key design features a dual transport layer that uses shared memory for fast on-host simulation and JPEG-based streaming for remote hardware, along with data-aware compression and Slurm-based evaluation tooling. VLAgents standardizes data exchange with a lightweight Obs/Act/Agent interface, supports multiple VLAs via a wrapper, and leverages RPc for remote calls to minimize serialization overhead. Empirical results show VLAgents outperforms model-specific servers in RTT across local and network deployments (up to 220 Hz network inference and ~0.3 ms transport delay), demonstrating substantial speedups (≈3x) due to its shared-memory and JPEG compression approach.

Abstract

The rapid emergence of Vision-Language-Action models (VLAs) has a significant impact on robotics. However, their deployment remains complex due to the fragmented interfaces and the inherent communication latency in distributed setups. To address this, we introduce VLAgents, a modular policy server that abstracts VLA inferencing behind a unified Gymnasium-style protocol. Crucially, its communication layer transparently adapts to the context by supporting both zero-copy shared memory for high-speed simulation and compressed streaming for remote hardware. In this work, we present the architecture of VLAgents and validate it by integrating seven policies -- including OpenVLA and Pi Zero. In a benchmark with both local and remote communication, we further demonstrate how it outperforms the default policy servers provided by OpenVLA, OpenPi, and LeRobot. VLAgents is available at https://github.com/RobotControlStack/vlagents

VLAgents: A Policy Server for Efficient VLA Inference

TL;DR

The paper tackles fragmentation and latency in Vision-Language-Action (VLA) robotics by introducing VLAgents, a model-agnostic policy server with a Gymnasium-like interface that bridges VLAs to both simulated and real robots. Its key design features a dual transport layer that uses shared memory for fast on-host simulation and JPEG-based streaming for remote hardware, along with data-aware compression and Slurm-based evaluation tooling. VLAgents standardizes data exchange with a lightweight Obs/Act/Agent interface, supports multiple VLAs via a wrapper, and leverages RPc for remote calls to minimize serialization overhead. Empirical results show VLAgents outperforms model-specific servers in RTT across local and network deployments (up to 220 Hz network inference and ~0.3 ms transport delay), demonstrating substantial speedups (≈3x) due to its shared-memory and JPEG compression approach.

Abstract

The rapid emergence of Vision-Language-Action models (VLAs) has a significant impact on robotics. However, their deployment remains complex due to the fragmented interfaces and the inherent communication latency in distributed setups. To address this, we introduce VLAgents, a modular policy server that abstracts VLA inferencing behind a unified Gymnasium-style protocol. Crucially, its communication layer transparently adapts to the context by supporting both zero-copy shared memory for high-speed simulation and compressed streaming for remote hardware. In this work, we present the architecture of VLAgents and validate it by integrating seven policies -- including OpenVLA and Pi Zero. In a benchmark with both local and remote communication, we further demonstrate how it outperforms the default policy servers provided by OpenVLA, OpenPi, and LeRobot. VLAgents is available at https://github.com/RobotControlStack/vlagents
Paper Structure (4 sections, 3 figures)

This paper contains 4 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Results and Conclusion

Figures (3)

  • Figure 1: The architecture of VLAgents. Each environment implements a thin wrapper that translates observations and actions into the expected data format (see \ref{['fig:interface']}). A central control loop takes the state and forwards it via either shared memory or a TCP connection with JPEG-encoding to the policy server. The policy server uses the interface from \ref{['fig:interface']} to run an inference pass with the corresponding model. This yields an action, which is then returned and used by the environment loop to step the environment to obtain the next state.
  • Figure 2: VLA policy interface.
  • Figure 3: Mean Round-Trip Time (RTT) for different policy servers with two $224\times224$ RGB cameras. Localhost indicates that the client and server are running on the same machine, while network indicates execution across different machines. For the network setting, the machines were connected in a Local Area Network with a 1 Gbit Ethernet connection.