Ranking-aware Reinforcement Learning for Ordinal Ranking

TL;DR

RARL tackles the challenge of learning both precise ordinal values and correct item rankings by unifying regression and L2R under a verifiable reward framework. It defines a regression reward and a ranking reward, combines them into a final reward with weights $\lambda_1$, $\lambda_2$, and $\lambda_3$, and uses a two-stage training strategy to stabilize learning. To address entropy collapse in policy optimization, it introduces the Response Mutation Operation (RMO) that replaces low-reward responses to reinject gradient signal. Empirical results on UTKFace, COCO-REM, and AVA show state-of-the-art performance, with ablations demonstrating the benefits of ranking-focused supervision and the RMO. This approach demonstrates effective, scalable ranking for vision-language tasks with ordinal outputs.

Abstract

Ordinal regression and ranking are challenging due to inherent ordinal dependencies that conventional methods struggle to model. We propose Ranking-Aware Reinforcement Learning (RARL), a novel RL framework that explicitly learns these relationships. At its core, RARL features a unified objective that synergistically integrates regression and Learning-to-Rank (L2R), enabling mutual improvement between the two tasks. This is driven by a ranking-aware verifiable reward that jointly assesses regression precision and ranking accuracy, facilitating direct model updates via policy optimization. To further enhance training, we introduce Response Mutation Operations (RMO), which inject controlled noise to improve exploration and prevent stagnation at saddle points. The effectiveness of RARL is validated through extensive experiments on three distinct benchmarks.

Figures (1)

• Figure 1: Flowchart of proposed RARL. For a given image-question pair, our RARL model generates multiple reasoning-based responses. It is then optimized via policy gradient using ranking-aware verifiable rewards. Then, the response mutation operation is employed to reactivate gradient signals and escape saddle points.