HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Models
Xin Xie, Jiaxian Guo, Dong Gong
TL;DR
HyperAlign presents a hypernetwork that generates low-rank adapters to dynamically modulate diffusion model denoising operators at test time, enabling efficient, trajectory-level alignment to human preferences. By predicting $\Delta\theta_t$ through a perception encoder and transformer decoder, HyperAlign offers three weight-generation strategies (HyperAlign-S, -I, -P) to balance accuracy and efficiency, and optimizes with a reward objective plus a preference-regularization term to mitigate reward hacking. It demonstrates superior semantic alignment and aesthetic quality on diffusion (SD) and flow-matching (FLUX) backbones, with practical inference times compared to gradient-based approaches. The approach is extensible to flow-matching models and supported by extensive ablations, user studies, and supplemental analyses, underscoring its potential to scale high-quality, user-aligned image synthesis while preserving diversity and realism.
Abstract
Diffusion models achieve state-of-the-art performance but often fail to generate outputs that align with human preferences and intentions, resulting in images with poor aesthetic quality and semantic inconsistencies. Existing alignment methods present a difficult trade-off: fine-tuning approaches suffer from loss of diversity with reward over-optimization, while test-time scaling methods introduce significant computational overhead and tend to under-optimize. To address these limitations, we propose HyperAlign, a novel framework that trains a hypernetwork for efficient and effective test-time alignment. Instead of modifying latent states, HyperAlign dynamically generates low-rank adaptation weights to modulate the diffusion model's generation operators. This allows the denoising trajectory to be adaptively adjusted based on input latents, timesteps and prompts for reward-conditioned alignment. We introduce multiple variants of HyperAlign that differ in how frequently the hypernetwork is applied, balancing between performance and efficiency. Furthermore, we optimize the hypernetwork using a reward score objective regularized with preference data to reduce reward hacking. We evaluate HyperAlign on multiple extended generative paradigms, including Stable Diffusion and FLUX. It significantly outperforms existing fine-tuning and test-time scaling baselines in enhancing semantic consistency and visual appeal.