FinPercep-RM: A Fine-grained Reward Model and Co-evolutionary Curriculum for RL-based Real-world Super-Resolution

TL;DR

The paper tackles reward hacking and training instability in Real-ISR by introducing FinPercep-RM, a diagnostic reward model that outputs a global quality score and a fine-grained Perceptual Degradation Map, trained on the 30k-sample FGR-30k dataset. To stabilize training while preserving robustness, it introduces Co-evolutionary Curriculum Learning (CCL), which progressively expands the RM’s capacity from a global predictor to a full fine-grained diagnostic head and co-evolves the Generator’s curriculum accordingly. Empirical results show improved perceptual quality and robustness across multiple Real-ISR baselines and RLHF strategies, with ablations highlighting the importance of combining pixel- and feature-level distinctions and of structured curricula. Overall, the approach offers a practical, robust framework for aligning Real-ISR with human perceptual preferences and reducing reward hacking in RLHF-guided SR tasks.

Abstract

Reinforcement Learning with Human Feedback (RLHF) has proven effective in image generation field guided by reward models to align human preferences. Motivated by this, adapting RLHF for Image Super-Resolution (ISR) tasks has shown promise in optimizing perceptual quality with Image Quality Assessment (IQA) model as reward models. However, the traditional IQA model usually output a single global score, which are exceptionally insensitive to local and fine-grained distortions. This insensitivity allows ISR models to produce perceptually undesirable artifacts that yield spurious high scores, misaligning optimization objectives with perceptual quality and results in reward hacking. To address this, we propose a Fine-grained Perceptual Reward Model (FinPercep-RM) based on an Encoder-Decoder architecture. While providing a global quality score, it also generates a Perceptual Degradation Map that spatially localizes and quantifies local defects. We specifically introduce the FGR-30k dataset to train this model, consisting of diverse and subtle distortions from real-world super-resolution models. Despite the success of the FinPercep-RM model, its complexity introduces significant challenges in generator policy learning, leading to training instability. To address this, we propose a Co-evolutionary Curriculum Learning (CCL) mechanism, where both the reward model and the ISR model undergo synchronized curricula. The reward model progressively increases in complexity, while the ISR model starts with a simpler global reward for rapid convergence, gradually transitioning to the more complex model outputs. This easy-to-hard strategy enables stable training while suppressing reward hacking. Experiments validates the effectiveness of our method across ISR models in both global quality and local realism on RLHF methods.

Figures (4)

• Figure 1: Motivation for FinPercep-RM and CCL. (a) Standard IQAs lack fine-grained perception and struggle to penalize local distortions, while Ours aligns with human judgment (User Study). (b) The training curves illustrate the stability-robustness dilemma: baseline IQA rewards (blue/purple) converge quickly, while FinPercep-RM (light blue) is oscillatory and unstable. Our complete method (FinPercep-RM w/ CCL, orange) achieves stable and optimal convergence. (c) Visualization of Reward Hacking: baseline rewards (W/ CLIP-IQA, W/ MAN-IQA) produce local artifacts, whereas ours results are faithful to the Ground Truth.
• Figure 2: The overall pipeline of the proposed FinPercep-RM and Co-evolutionary Curriculum Learning (CCL) framework. FinPercep-RM produces a Fine-grained Perceptual Degradation Map that captures spatially localized defect likelihood and intensity, and the reward model is progressively expanded from small-variance, coarse-grained rewards to large-variance, fine-grained signals. During training, under the CCL mechanism, the Generator first learns with the coarse global reward from $RM_0$ for a stable and easy initialization, and is then co-evolutionarily guided by increasingly strict $RM_k$ versions to enhance local fidelity and suppress reward hacking.
• Figure 3: FGR-30k construction pipeline. We synthesize fine-grained distortion samples by swapping artifact-rich regions from diffusion-based SR outputs into clean images, using both random and semantic masks. Ground-truth perceptual degradation maps are generated by fusing pixel- and feature-level dissimilarities (via DINOv3), providing spatially precise supervision for training FinPercep-RM.
• Figure 4: Qualitative comparisons with state-of-the-art Real-ISR methods on on RealSR based on RLHF method of REFL xu2023imagereward.