Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPOGW: a Score-based Preference Optimization method via Group-Wise comparison for workflows

Yitong Cui, Liu Liu, Baosheng Yu, Jiayan Qiu, Xikai Zhang, Likang Xiao, Yixing Liu, Quan Chen

TL;DR

SPOGW tackles the automation of agentic workflows by enabling score-based optimization in continuous space through group-wise comparisons of candidate workflows. It combines Iterative offline GRPO (ioGRPO) to decouple data collection from policy updates with an Advantage-Masked KL (mKL) that concentrates regularization on advantageous responses, yielding stable and efficient learning. Across five benchmarks spanning math, coding, and QA, SPOGW achieves state-of-the-art results and demonstrates that group-wise data processing, offline data collection, and targeted KL penalties synergistically improve performance, scalability, and robustness. The approach reduces reliance on pairwise preferences and scales to diverse domains, offering a practical path toward automated generation and optimization of complex workflows.

Abstract

Large language models (LLMs) have exhibited significant capabilities in addressing challenging problems throughout various fields, often through the use of agentic workflows that adhere to structured instructions and multi-step procedures. However, designing such workflows demands substantial manual effort, posing challenges to scalability and generalizability. Recent studies have aimed to minimize the human intervention needed for their construction, leading to advances in automated techniques for optimizing agentic workflows. However, current approaches are often constrained by their limited representational capacity, insufficient adaptability, weak scalability, and pairwise comparison paradigm -- issues that stem primarily from a dependence on discrete optimization techniques. To overcome these limitations, we introduce a new score-based preference approach, refereed as SPOGW, which operates directly on cardinal reward signals through group-wise comparison and enables more efficient and stable optimization in a continuous space. SPOGW incorporates Iterative offline GRPO (ioGRPO) with advantage-masked KL divergence (mKL), which regulates training update by placing greater emphasis on the advantageous regions of the policy response. In five benchmark datasets covering mathematical reasoning, coding, and question answering, SPOGW matches or exceeds the performance of current state-of-the-art approaches, presenting a viable and forward-looking methodology for automated generation and optimization of agentic workflows.

SPOGW: a Score-based Preference Optimization method via Group-Wise comparison for workflows

TL;DR

SPOGW tackles the automation of agentic workflows by enabling score-based optimization in continuous space through group-wise comparisons of candidate workflows. It combines Iterative offline GRPO (ioGRPO) to decouple data collection from policy updates with an Advantage-Masked KL (mKL) that concentrates regularization on advantageous responses, yielding stable and efficient learning. Across five benchmarks spanning math, coding, and QA, SPOGW achieves state-of-the-art results and demonstrates that group-wise data processing, offline data collection, and targeted KL penalties synergistically improve performance, scalability, and robustness. The approach reduces reliance on pairwise preferences and scales to diverse domains, offering a practical path toward automated generation and optimization of complex workflows.

Abstract

Large language models (LLMs) have exhibited significant capabilities in addressing challenging problems throughout various fields, often through the use of agentic workflows that adhere to structured instructions and multi-step procedures. However, designing such workflows demands substantial manual effort, posing challenges to scalability and generalizability. Recent studies have aimed to minimize the human intervention needed for their construction, leading to advances in automated techniques for optimizing agentic workflows. However, current approaches are often constrained by their limited representational capacity, insufficient adaptability, weak scalability, and pairwise comparison paradigm -- issues that stem primarily from a dependence on discrete optimization techniques. To overcome these limitations, we introduce a new score-based preference approach, refereed as SPOGW, which operates directly on cardinal reward signals through group-wise comparison and enables more efficient and stable optimization in a continuous space. SPOGW incorporates Iterative offline GRPO (ioGRPO) with advantage-masked KL divergence (mKL), which regulates training update by placing greater emphasis on the advantageous regions of the policy response. In five benchmark datasets covering mathematical reasoning, coding, and question answering, SPOGW matches or exceeds the performance of current state-of-the-art approaches, presenting a viable and forward-looking methodology for automated generation and optimization of agentic workflows.

Paper Structure

This paper contains 22 sections, 6 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Reinforcement Learning for Advanced Reasoning in LLMs
  4. Automated Agentic Workflow Optimization
  5. Agentic Workflows
  6. Automated Workflow Optimization
  7. Methods
  8. Score-based Preference Dataset
  9. Data construction
  10. Filtering and Screening
  11. Group Sharpening
  12. From GRPO to Iterative offline GRPO
  13. Advantage-Masked KL Restriction
  14. Experiments
  15. Experimental Setup
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Pipeline of SPOGW: The framework generates multiple workflows for each query, then executes and evaluates each workflow to obtain a score, and then conducts combination and subsequent group-wise data processing, which feeds into ioGRPO optimization cycle.
  • Figure 2: Analysis of dataset characteristics under different processing methods shows that $\mathcal{D}_\text{SS}$ achieves superior variance and clearer quality separation. The training group size is fixed at 8. Median Interval Length (MIL) is the gap between the 4th and 5th highest scores.
  • Figure 3: Analysis of the KL coefficient $\beta$, group size $2t$ and dataset size $d$ on HumanEval.