BinaryPPO: Efficient Policy Optimization for Binary Classification

TL;DR

BinaryPPO introduces an offline reinforcement learning framework that reframes binary classification as a confidence-aware decision problem for LLMs. It defines a probabilistic reward $r(x,a,y)=\kappa \cdot s(a,y) \cdot f(π_{θ_{old}}(a|x))$ with $f(p)=\log(p)$ and optimizes a PPO-style objective with a learned value function to stabilize training. Across eight benchmarks and two base models, BinaryPPO achieves substantial gains over SFT and PPO baselines, with high in-domain accuracy (up to $\approx 99\%$) and meaningful out-of-distribution generalization, while preserving safety-related normative properties. The approach demonstrates that confidence-weighted reward design can be a robust alternative to supervised fine-tuning for noisy, imbalanced binary tasks and suggests extensions to multi-class settings.

Abstract

Supervised fine-tuning (SFT) is the standard approach for binary classification tasks such as toxicity detection, factuality verification, and causal inference. However, SFT often performs poorly in real-world settings with label noise, class imbalance, or sparse supervision. We introduce BinaryPPO, an offline reinforcement learning large language model (LLM) framework that reformulates binary classification as a reward maximization problem. Our method leverages a variant of Proximal Policy Optimization (PPO) with a confidence-weighted reward function that penalizes uncertain or incorrect predictions, enabling the model to learn robust decision policies from static datasets without online interaction. Across eight domain-specific benchmarks and multiple models with differing architectures, BinaryPPO improves accuracy by 40-60 percentage points, reaching up to 99%, substantially outperforming supervised baselines. We provide an in-depth analysis of the role of reward shaping, advantage scaling, and policy stability in enabling this improvement. Overall, we demonstrate that confidence-based reward design provides a robust alternative to SFT for binary classification. Our code is available at https://github.com/psyonp/BinaryPPO.

Figures (6)

• Figure 1: By framing classification as a binary decision problem and optimizing a confidence-based reward function, BinaryPPO achieves consistent performance improvements compared to PPO across interdisciplinary benchmarks.
• Figure 2: High-level schematic diagram of BinaryPPO. The policy $\pi_\theta$ proposes actions, which are evaluated via confidence-weighted rewards $r(a, y)$ and a value function $V_\phi(h)$; defining the composite loss $\mathcal{L}$ that drives consistent binary performance.
• Figure 3: Effect of instilled confidence on post-training behavior under BinaryPPO. (a) Accuracy increases monotonically with confidence, indicating alignment between confidence and correctness. (b) Advantage is maximized at intermediate confidence, suggesting the strongest learning signal occurs for moderately confident predictions (0.6-0.8).
• Figure 4: Representative training curves for Qwen 2.5–3B and Gemma 2–2B aggregated across datasets, demonstrating stable fine-tuning behavior under BinaryPPO.
• Figure 5: Evolution of policy metrics across epochs. (Top) Policy entropy converges within seven epochs, with Qwen maintaining higher entropy than Gemma. (Bottom) Approximate KL divergence between successive policies decreases and stabilizes within the first five epochs, indicating consistent policy refinement.