When Rubrics Fail: Error Enumeration as Reward in Reference-Free RL Post-Training for Virtual Try-On

TL;DR

This work introduces Cascaded Error Counting (CEC) as an evaluation metric, and curate Mismatch-DressCode, a benchmark with maximal attribute mismatch to stress-test reward designs, and suggests that when ideal answers are unavailable, counting errors provide a stronger signal than constructing rubrics.

Abstract

Reinforcement learning with verifiable rewards (RLVR) and Rubrics as Rewards (RaR) have driven strong gains in domains with clear correctness signals and even in subjective domains by synthesizing evaluation criteria from ideal reference answers. But many real-world tasks admit multiple valid outputs and lack the single ideal answer that rubric generation depends on. We identify this reference-free setting as a gap in current post-training methods and propose Implicit Error Counting (IEC) to fill it. Instead of checking what a response gets right against a rubric, IEC enumerates what it gets wrong, applying severity-weighted scores across task-relevant axes and converting them into calibrated per-aspect rewards. We show that naïve explicit enumeration is too noisy for stable optimization, and that two design choices: implicit score emission and group calibration are necessary to make error counting a reliable reward. As a case study, we validate IEC on virtual try-on (VTO), a domain that is simultaneously too constrained for holistic scoring and too permissive for rubric-based evaluation: subtle garment errors are unacceptable, yet many output variations are correct. We introduce Cascaded Error Counting (CEC) as an evaluation metric, which tracks human preferences well (60% top-1 vs. 30% others), and curate Mismatch-DressCode (MDressBench), a benchmark with maximal attribute mismatch to stress-test reward designs. On MDressBench, IEC outperforms RaR across all metrics (CEC: 5.31 vs. 5.60 on flat references; 5.20 vs. 5.53 on non-flat). On VITON-HD and DressCode, IEC matches or surpasses six baselines on 6 of 8 perceptual metrics. These results suggest that when ideal answers are unavailable, counting errors provide a stronger signal than constructing rubrics.

Figures (17)

• Figure 1: Qualitative comparison of reward designs on non-flat references: Given the input, IEC avoids attribute preservation errors across 3 examples, garment transfer failures in example C, source integrity issues in A and D, and realism errors in examples B and C, over all other reward designs. Per-category radar plots are in Appendix \ref{['supp:radar']}.
• Figure 2: Training dynamics for four reward designs during GRPO post-training. Left 2$\times$2: Per-method curves showing raw rewards (faded) and EMA (solid, $\alpha=0.99$). Right: Normalized EMA showing relative improvement. IEC achieves smooth, monotonic improvement. Direct and RaR show high variance. EEC regresses early, confirming that explicit enumeration destabilizes optimization.
• Figure 3: MDressBench Examples.Flat references (left) show the garment as a lay-flat product image on a neutral background. Non-flat references (right) show the garment worn by a model, requiring the model to "extract" the garment from arbitrary poses and backgrounds.
• Figure 4: Effect of Best-of-$N$ with 12 generations. Higher $N$ typically reduces reward diversity and performance.
• Figure 5: Training dynamics. CEC $\downarrow$ for IEC and RaR across post-training iterations. IEC consistently achieves lower error.