Understanding Reward Hacking in Text-to-Image Reinforcement Learning

TL;DR

The paper addresses reward hacking in text-to-image reinforcement learning by showing that common proxy rewards (aesthetic/human preference, and prompt–image consistency) can be exploited, producing artifact-laden images with high scores. It systematically analyzes how different reward types drive biased behaviors and reveals a universal pattern of artifact generation that generalizes across setups. To mitigate this, it introduces ArtifactReward, an artifact-aware supplementary reward learned via automatic prompt optimization, defined as $R_{\text{ArtifactReward}} = \frac{1}{1+ e^{[\log(p_{\text{yes}}) - \log(p_{\text{no}})]}}$, which penalizes artifact-containing outputs. Incorporating ArtifactReward into various T2I RL pipelines consistently improves realism, reduces artifacts, and enhances cross-benchmark performance on WISE, LLM4LLM, and EvalAlign, demonstrating the practicality of lightweight reward augmentation for safer, more human-aligned image generation. Overall, the work provides a principled pathway to strengthen reward signals in multimodal RL by combining targeted artifact detection with automatic prompt optimization, yielding more robust and faithful T2I generation.

Abstract

Reinforcement learning (RL) has become a standard approach for post-training large language models and, more recently, for improving image generation models, which uses reward functions to enhance generation quality and human preference alignment. However, existing reward designs are often imperfect proxies for true human judgment, making models prone to reward hacking--producing unrealistic or low-quality images that nevertheless achieve high reward scores. In this work, we systematically analyze reward hacking behaviors in text-to-image (T2I) RL post-training. We investigate how both aesthetic/human preference rewards and prompt-image consistency rewards individually contribute to reward hacking and further show that ensembling multiple rewards can only partially mitigate this issue. Across diverse reward models, we identify a common failure mode: the generation of artifact-prone images. To address this, we propose a lightweight and adaptive artifact reward model, trained on a small curated dataset of artifact-free and artifact-containing samples. This model can be integrated into existing RL pipelines as an effective regularizer for commonly used reward models. Experiments demonstrate that incorporating our artifact reward significantly improves visual realism and reduces reward hacking across multiple T2I RL setups, demonstrating the effectiveness of lightweight reward augment serving as a safeguard against reward hacking.

Figures (14)

• Figure 1: Evolution of metrics over training steps trained on Janus-Pro-1B chen2025janus. Blue color denotes the model trained with HPS wu2023human; Orange color denotes the model trained with GDino liu2024grounding; Green color denotes the model trained with HPS and GDino; Red color denotes the model trained with finetuned ORM guo2025can.
• Figure 2: Images generated with prompt "Traditional food of the Mid-Autumn Festival" in WISE niu2025wise benchmark under different training reward configurations. This prompt expect the image to show mooncakes.
• Figure 3: Images generated with prompt "a photo of a cow" in LLM4LLM wang2025lmm4lmm benchmark under different training reward configurations.
• Figure 4: Evolution of metrics over training steps with different categories of prompts trained on Janus-Pro-7B chen2025janus. Blue color denotes the model trained with HPS wu2023human; Orange color denotes the model trained with GDino liu2024grounding; Green color denotes the model trained with HPS and GDino; Red color denotes the model trained with finetuned ORM guo2025can.
• Figure 5: Performance on WISE niu2025wise benchmark across different categories trained on Janus-Pro-1B chen2025janus.