Scaling Image and Video Generation via Test-Time Evolutionary Search

TL;DR

EvoSearch introduces a generalist, training-free test-time scaling framework that optimizes image and video generation by evolving samples along the denoising trajectory in diffusion and flow models. By transforming flow sampling to a stochastic process and applying evolution-inspired selection and mutation, EvoSearch actively explores high-reward regions beyond fixed candidate pools, outperforming Best-of-N and particle sampling across NFEs. Empirical results show substantial gains on both diffusion-based and flow-based architectures for image and video tasks, improved alignment with unseen rewards, and competitive performance of smaller models against larger ones under increased inference budgets. The approach offers a practical, model-agnostic pathway to leverage inference-time computation for substantial quality and diversity improvements, with noted avenues for future enhancements in mutation design and interpretability.

Abstract

As the marginal cost of scaling computation (data and parameters) during model pre-training continues to increase substantially, test-time scaling (TTS) has emerged as a promising direction for improving generative model performance by allocating additional computation at inference time. While TTS has demonstrated significant success across multiple language tasks, there remains a notable gap in understanding the test-time scaling behaviors of image and video generative models (diffusion-based or flow-based models). Although recent works have initiated exploration into inference-time strategies for vision tasks, these approaches face critical limitations: being constrained to task-specific domains, exhibiting poor scalability, or falling into reward over-optimization that sacrifices sample diversity. In this paper, we propose \textbf{Evo}lutionary \textbf{Search} (EvoSearch), a novel, generalist, and efficient TTS method that effectively enhances the scalability of both image and video generation across diffusion and flow models, without requiring additional training or model expansion. EvoSearch reformulates test-time scaling for diffusion and flow models as an evolutionary search problem, leveraging principles from biological evolution to efficiently explore and refine the denoising trajectory. By incorporating carefully designed selection and mutation mechanisms tailored to the stochastic differential equation denoising process, EvoSearch iteratively generates higher-quality offspring while preserving population diversity. Through extensive evaluation across both diffusion and flow architectures for image and video generation tasks, we demonstrate that our method consistently outperforms existing approaches, achieves higher diversity, and shows strong generalizability to unseen evaluation metrics. Our project is available at the website https://tinnerhrhe.github.io/evosearch.

Figures (27)

• Figure 1: We propose Evolutionary Search (EvoSearch), a novel and generalist test-time scaling framework applicable to both image and video generation tasks. EvoSearch significantly enhances sample quality through strategic computation allocation during inference, enabling Stable Diffusion 2.1 to exceed GPT4o, and Wan 1.3B to outperform Wan 14B model and Hunyuan 13B model with $10\times$ fewer parameters.
• Figure 2: Visualization of a test-time alignment experiment. We train a diffusion model with $3$-layer MLP on Gaussian mixtures (pre-trained distribution), with the goal to capture multimodal unseen target distribution, where reward $r(X,Y)=-|X^2+Y^2-4|$. EvoSearch achieves superior performance, capturing all the modes with the highest reward (-0.74).
• Figure 3: Overview of our method. EvoSearch progressively moves forward along the denoising trajectory to refine and explore new states.
• Figure 4: t-SNE Visualization of latent $\bm{x}_t$ from SD2.1 model at different steps, colored by their corresponding ImageReward scores. At denoising step 0, $\bm{x}_t$ is Gaussian noises. Top row: Results from Stable Diffusion 2.1 model. Bottom row: Results from Flux.1-dev.
• Figure 5: Scaling behavior of EvoSearch and baselines as inference-time computation increases on DrawBench. Top: SD2.1. Bottom: Flux.1-dev. (a) and (b) use ImageReward and ClipScore as guidance rewards, respectively.