SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models

TL;DR

This work tackles post-training alignment of video generation with human preferences by addressing annotation noise, reward hacking, and limited RM architectures. It introduces SoliReward, combining low-noise single-item binary annotations, cross-prompt pairing, and the BT-WT loss with the HPQA reward-adapter to produce robust, multi-layered reward signals. Empirical results show stronger RM accuracy on in-domain and out-of-domain data and improved post-training performance when guiding video generators, with reduced vulnerability to reward hacking. The contributions include new data collection and labeling strategies, a novel loss function to regularize positive scores, and a hierarchical attention-based RM architecture, validated on physical plausibility, subject deformity, and semantic alignment benchmarks.

Abstract

Post-training alignment of video generation models with human preferences is a critical goal. Developing effective Reward Models (RMs) for this process faces significant methodological hurdles. Current data collection paradigms, reliant on in-prompt pairwise annotations, suffer from labeling noise. Concurrently, the architectural design of VLM-based RMs, particularly their output mechanisms, remains underexplored. Furthermore, RM is susceptible to reward hacking in post-training. To mitigate these limitations, we propose SoliReward, a systematic framework for video RM training. Our framework first sources high-quality, cost-efficient data via single-item binary annotations, then constructs preference pairs using a cross-prompt pairing strategy. Architecturally, we employ a Hierarchical Progressive Query Attention mechanism to enhance feature aggregation. Finally, we introduce a modified BT loss that explicitly accommodates win-tie scenarios. This approach regularizes the RM's score distribution for positive samples, providing more nuanced preference signals to alleviate over-focus on a small number of top-scoring samples. Our approach is validated on benchmarks evaluating physical plausibility, subject deformity, and semantic alignment, demonstrating improvements in direct RM evaluation metrics and in the efficacy of post-training on video generation models. Code and benchmark will be publicly available.

Figures (9)

• Figure 1: Pipeline of SoliReward, our framework for data annotation and training of video reward models. (a) We introduce a single-item binary annotation method, coupled with a cross-prompt pairing strategy, to mitigate annotation noise. Furthermore, to alleviate reward hacking, we propose the Bradley-Terry with Win-Tie (BT-WT) loss. (b/c) We propose a novel VLM-based Reward Model (VLM-RM) architecture, featuring a Hierarchical Progressive Query Attention (HPQA) adapter. This adapter progressively aggregates multi-level representations from the VLM backbone to compute a robust reward score.
• Figure 2: The reward score distributions on the semantic alignment task reveal that alternative architectures suffer from severe score clustering, assigning identical ratings to many samples.
• Figure 3: Reward distribution for BT and BT-WT. BT-WT contributes to a more concentrated distribution in the high-score segment for positive samples.
• Figure 4: Intra-group advantage distribution in BT-WT exhibits smaller advantages for top-ranked samples compared to BT, thereby mitigating the over-optimization.
• Figure 5: Visual results guided by different reward models. From top to bottom: Baseline (HunyuanVideo), VideoAlign, and Ours.