Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards

Zehao Li, Zhenyu Wu, Yibo Zhao, Bowen Yang, Jingjing Xie, Zhaoyang Liu, Zhoumianze Liu, Kaiming Jin, Jianze Liang, Zonglin Li, Feng Wu, Bowen Zhou, Zun Wang, Zichen Ding

Abstract

Reinforcement Learning (RL) has the potential to improve the robustness of GUI agents in stochastic environments, yet training is highly sensitive to the quality of the reward function. Existing reward approaches struggle to achieve both scalability and performance. To address this, we propose OS-Themis, a scalable and accurate multi-agent critic framework. Unlike a single judge, OS-Themis decomposes trajectories into verifiable milestones to isolate critical evidence for decision making and employs a review mechanism to strictly audit the evidence chain before making the final verdict. To facilitate evaluation, we further introduce OmniGUIRewardBench (OGRBench), a holistic cross-platform benchmark for GUI outcome rewards, where all evaluated models achieve their best performance under OS-Themis. Extensive experiments on AndroidWorld show that OS-Themis yields a 10.3% improvement when used to support online RL training, and a 6.9% gain when used for trajectory validation and filtering in the self-training loop, highlighting its potential to drive agent evolution.

OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards

Abstract

Reinforcement Learning (RL) has the potential to improve the robustness of GUI agents in stochastic environments, yet training is highly sensitive to the quality of the reward function. Existing reward approaches struggle to achieve both scalability and performance. To address this, we propose OS-Themis, a scalable and accurate multi-agent critic framework. Unlike a single judge, OS-Themis decomposes trajectories into verifiable milestones to isolate critical evidence for decision making and employs a review mechanism to strictly audit the evidence chain before making the final verdict. To facilitate evaluation, we further introduce OmniGUIRewardBench (OGRBench), a holistic cross-platform benchmark for GUI outcome rewards, where all evaluated models achieve their best performance under OS-Themis. Extensive experiments on AndroidWorld show that OS-Themis yields a 10.3% improvement when used to support online RL training, and a 6.9% gain when used for trajectory validation and filtering in the self-training loop, highlighting its potential to drive agent evolution.
Paper Structure (57 sections, 15 equations, 14 figures, 12 tables)

This paper contains 57 sections, 15 equations, 14 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. OS-Themis
  4. Overview
  5. Milestone Verification Module
  6. Verdict Calibration Module
  7. Experiments
  8. OmniGUIRewardBench
  9. Benchmark Construction.
  10. Main Results.
  11. Online RL
  12. Online RL Infrastructure.
  13. Task Design.
  14. Training Setup.
  15. Comparisons.
  16. ...and 42 more sections

Figures (14)

  • Figure 1: Limitations of existing approaches for reward modeling in GUI environments.
  • Figure 2: Overview of OS-Themis Framework. The framework primarily consists of two modules: the Milestone Verification Module (MVM) and the Verdict Calibration Module (VCM). For a GUI trajectory containing several <screenshot,think,action> steps, the Selector Agent in the MVM extracts key steps as milestones, which are then assigned binary scores by the Verifier Agent. Subsequently, the Reviewer Agent in the VCM continuously interacts with the MVM to ensure the rationality and completeness of the milestones, while the Judge Agent conducts the final scoring of the trajectory based on all information exchanged between the modules.
  • Figure 3: The performance of Qwen3-VL-4B under online RL scaling with OS-Themis, including mean reward growth and corresponding AndroidWorld accuracy across different training scales.
  • Figure 4: The performance of Qwen3-VL-4B and Qwen3-VL-8B on AndroidWorld after SFT using filtered data (the parentheses indicate the filtering method; All Data means the data is unfiltered).
  • Figure 5: Data Distribution of OGRBench
  • ...and 9 more figures