GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization

TL;DR

GAPO tackles the challenge of enforcing fine-grained constraints in large language models by marrying Generative Adversarial Networks with Proximal Policy Optimization, guided by an encoder-only reward model. The framework introduces constraint-aware data augmentation and an adversarial training loop where a generator improves outputs under progressively harder constraints while a reward model learns to discriminate compliant outputs. Empirical results show GAPO outperforms PPO, DPO, KTO, and ORPO, especially in scenarios demanding nuanced constraint understanding, and preferential prompting consistently yields stronger gains than preferential responses. The work advances robust, data-efficient control of LLM outputs with scalable training dynamics and provides a dedicated PDD dataset and evaluation setup to stress constraint adherence. Practical impact lies in enabling more reliable, interpretable constrained generation for applications like legal, medical, and automated documentation.

Abstract

Recent advances in large language models have highlighted the critical need for precise control over model outputs through predefined constraints. While existing methods attempt to achieve this through either direct instruction-response synthesis or preferential response optimization, they often struggle with constraint understanding and adaptation. This limitation becomes particularly evident when handling fine-grained constraints, leading to either hallucination or brittle performance. We introduce Generative Adversarial Policy Optimization (GAPO), a novel framework that combines GAN-based training dynamics with an encoder-only reward model to progressively learn and adapt to increasingly complex constraints. GAPO leverages adversarial training to automatically generate training samples of varying difficulty while utilizing the encoder-only architecture to better capture prompt-response relationships. Extensive experiments demonstrate GAPO's superior performance across multiple benchmarks, particularly in scenarios requiring fine-grained constraint handling, where it significantly outperforms existing methods like PPO, DPO, and KTO. Our results suggest that GAPO's unique approach to preferential prompt learning offers a more robust and effective solution for controlling LLM outputs. Code is avaliable in https://github.com/MikeGu721/GAPO.

Figures (5)

• Figure 1: Illustration of the procedural differences between Preferential Response and Preferential Prompt, emphasizing their distinct utilization of prompts and responses.
• Figure 2: The GAPO framework encompasses two distinct tuning phases. The initial phase consists of a warm-up period, during which the Reward Model is trained utilizing existing preference data. The subsequent phase implements adversarial training through a dual mechanism: the Generator is updated based on feedback from the Reward Model. The Reward Model undergoes training using a combination of Generator-produced data and existing preference data.
• Figure 3: Analysis of Correlative Factors Influencing GAPO's Performance on PDD and IFEval Benchmarks. The analysis utilizes 300 randomly sampled instances from the PDD test set and the complete IFEval test set with 108 samples for comprehensive evaluation.
• Figure 4: Detailed Performance Analysis Across Sequential Adversarial Training Stages. W indicates the warmup phase, and A represents the adversarial phase with alternating training between Generator and Reward Model components.
• Figure 5: Case study of model performance under different training baslines.