Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation

Zezhou Wang, Ziyun Zhang, Xiaoyi Zhang, Zhuzhong Qian, Yan Lu

TL;DR

BEPA tackles the challenge of leveraging a small pool of expert GUI trajectories to train end-to-end GUI policies under RLVR. It introduces LEVEL-1 self-rolled execution and LEVEL-2 self-aligned off-policy cache integrated with GRPO to align expert signals with the learner's policy. On OSWorld-Verified and cross-domain benchmarks, BEPA yields substantial improvements over pure GRPO and static expert integration, including a jump from 22.87% to 32.13% overall success and 5.74% to 10.30% on held-out data. The results highlight the value of dynamic, policy-consistent guidance for bridging the gap between framework-based experts and end-to-end GUI policies, with practical implications for scalable GUI agent training.

Abstract

Vision-language models are increasingly deployed as computer-use agents (CUAs) that operate desktops and browsers. Top-performing CUAs are framework-based systems that decompose planning and execution, while end-to-end screenshot-to-action policies are easier to deploy but lag behind on benchmarks such as OSWorld-Verified. GUI datasets like OSWorld pose two bottlenecks: they expose only a few hundred interactive, verifiable tasks and environments, and expert trajectories must be gathered by interacting with these environments, making such data hard to scale. We therefore ask how reinforcement learning from verifiable rewards (RLVR) can best exploit a small pool of exist expert trajectories to train end-to-end policies. Naively mixing these off-policy traces into on-policy RLVR is brittle: even after format conversion, expert trajectories exhibit structural mismatch and distribution shift from the learner. We propose BEPA (Bi-Level Expert-to-Policy Assimilation), which turns static expert traces into policy-aligned guidance via self-rolled reachable trajectories under the base policy (LEVEL-1) and a per-task, dynamically updated cache used in RLVR (LEVEL-2). On OSWorld-Verified, BEPA improves UITARS1.5-7B success from 22.87% to 32.13% and raises a held-out split from 5.74% to 10.30%, with consistent gains on MMBench-GUI and Online-Mind2Web. Our code and data are available at: https://github.com/LEON-gittech/Verl_GUI.git

From Off-Policy to On-Policy: Enhancing GUI Agents via Bi-level Expert-to-Policy Assimilation

TL;DR

BEPA tackles the challenge of leveraging a small pool of expert GUI trajectories to train end-to-end GUI policies under RLVR. It introduces LEVEL-1 self-rolled execution and LEVEL-2 self-aligned off-policy cache integrated with GRPO to align expert signals with the learner's policy. On OSWorld-Verified and cross-domain benchmarks, BEPA yields substantial improvements over pure GRPO and static expert integration, including a jump from 22.87% to 32.13% overall success and 5.74% to 10.30% on held-out data. The results highlight the value of dynamic, policy-consistent guidance for bridging the gap between framework-based experts and end-to-end GUI policies, with practical implications for scalable GUI agent training.

Abstract

Vision-language models are increasingly deployed as computer-use agents (CUAs) that operate desktops and browsers. Top-performing CUAs are framework-based systems that decompose planning and execution, while end-to-end screenshot-to-action policies are easier to deploy but lag behind on benchmarks such as OSWorld-Verified. GUI datasets like OSWorld pose two bottlenecks: they expose only a few hundred interactive, verifiable tasks and environments, and expert trajectories must be gathered by interacting with these environments, making such data hard to scale. We therefore ask how reinforcement learning from verifiable rewards (RLVR) can best exploit a small pool of exist expert trajectories to train end-to-end policies. Naively mixing these off-policy traces into on-policy RLVR is brittle: even after format conversion, expert trajectories exhibit structural mismatch and distribution shift from the learner. We propose BEPA (Bi-Level Expert-to-Policy Assimilation), which turns static expert traces into policy-aligned guidance via self-rolled reachable trajectories under the base policy (LEVEL-1) and a per-task, dynamically updated cache used in RLVR (LEVEL-2). On OSWorld-Verified, BEPA improves UITARS1.5-7B success from 22.87% to 32.13% and raises a held-out split from 5.74% to 10.30%, with consistent gains on MMBench-GUI and Online-Mind2Web. Our code and data are available at: https://github.com/LEON-gittech/Verl_GUI.git
Paper Structure (90 sections, 1 theorem, 21 equations, 28 figures, 12 tables, 2 algorithms)

This paper contains 90 sections, 1 theorem, 21 equations, 28 figures, 12 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. GUI Agents
  4. Framework-based agents.
  5. End-to-end agents.
  6. Reinforcement Learning for LLM Agents
  7. Preliminaries
  8. Bi-Level Expert-to-Policy Assimilation
  9. LEVEL-1: Self-Rolled Execution
  10. LEVEL-2: Off-Policy Assimilation
  11. Self-Aligned Off-Policy Cache.
  12. Dynamic Updates.
  13. Integration with GRPO
  14. Experiments
  15. Experiment Setup
  16. ...and 75 more sections

Key Result

Theorem 1

Let $\{\theta_k\}_{k=0}^{K-1}$ be the parameter sequence generated by BEPA with learning rate $\alpha_k = c/\sqrt{K}$. Under assumptions (i)–(iii), there exist constants $C_1,C_2>0$ such that where $\theta^*$ is an optimal solution of the surrogate objective. Thus BEPA converges to a first-order stationary point of $\mathcal{J}_{\mathrm{BEPA}}(\theta)$ at rate $O(1/\sqrt{K})$.

Figures (28)

  • Figure 1: BEPA overview. We exploit strong but mismatched experts via two modular, plug-and-play stages. LEVEL-1 initializes a policy-compatible guidance seed by re-rolling expert plans under the base policy. LEVEL-2 maintains a self-aligned per-task cache using the agent's own emerging successes, keeping the off-policy guidance within a controllable distribution gap relative to the evolving on-policy manifold. The cached guidance is injected into GRPO only upon total on-policy failure (i.e., when all rollouts in a group fail).
  • Figure 2: Distribution bias from framework experts to end-to-end GUI policies.Top: action-space mismatch (Agent S2 includes non-convertible framework actions beyond UI-TARS primitives). Bottom: inference structure mismatch. Agent S2 traces (left) interleave multi-role artifacts (planner/executor/grounder), yielding low likelihood under the base policy (low $\log \pi_\theta(\tau)$). Naive conversion (middle) maps format but largely preserves the framework structure, remaining off-manifold. Self-rolled execution (right) re-generates actions under the base policy (plan-conditioned), producing more policy-compatible, higher-likelihood trajectories.
  • Figure 3: Token-probability distributions and simple diagnostics on $\mathcal{D}_{\mathrm{upd}}$.Top: Raw Agent S2 traces (gray) versus converted traces (green). Conversion substantially reduces the extreme low-probability tail ($p<0.2$) but still leaves a large mass in regions atypical for the base policy. Middle: LEVEL-1 self-rolled guidance (orange) and LEVEL-2 cache-updated traces (purple). LEVEL-1 makes guidance highly reachable (peaked near $p\approx 1$), while LEVEL-2 reshapes the cache toward the on-policy reference, yielding a much smaller JS divergence. Bottom: On-policy baseline. Right: Tail mass $\Pr(p<0.2)$ and JS w.r.t. the on-policy reference.
  • Figure 4: Dynamic environments in OSWorld. (1) Pop-ups: human-verification challenges and pop-up dialogs appear stochastically with the random content. (2) Web Content: Real time content and random ads.
  • Figure 5: Training dynamics. (a) Policy entropy: SFT+RL collapses during the SFT stage, and LEVEL-1 shows a pronounced entropy drop due to high-confidence plan-conditioned self-rolled guidance; in contrast, LUFFY/Replacement maintain entropy close to the on-policy baseline, while BEPA sustains a moderate entropy profile alongside steadily improving expert-only success. (b--c) Avg. log-probability on successful trajectories: BEPA improves off-policy and on-policy successes in tandem, indicating gradual assimilation without degrading on-policy learning; LUFFY/Replacement exhibit an early dip and only partial recovery on on-policy log-probability, converging to a low plateau, while SFT+RL shows overfitting that harms on-policy improvement.
  • ...and 23 more figures

Theorems & Definitions (1)

  • Theorem 1: Convergence of BEPA