Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Naïve PAINE: Lightweight Text-to-Image Generation Improvement with Prompt Evaluation

Joong Ho Kim, Nicholas Thai, Souhardya Saha Dip, Dong Lao, Keith G. Mills

Abstract

Text-to-Image (T2I) generation is primarily driven by Diffusion Models (DM) which rely on random Gaussian noise. Thus, like playing the slots at a casino, a DM will produce different results given the same user-defined inputs. This imposes a gambler's burden: To perform multiple generation cycles to obtain a satisfactory result. However, even though DMs use stochastic sampling to seed generation, the distribution of generated content quality highly depends on the prompt and the generative ability of a DM with respect to it. To account for this, we propose Naïve PAINE for improving the generative quality of Diffusion Models by leveraging T2I preference benchmarks. We directly predict the numerical quality of an image from the initial noise and given prompt. Naïve PAINE then selects a handful of quality noises and forwards them to the DM for generation. Further, Naïve PAINE provides feedback on the DM generative quality given the prompt and is lightweight enough to seamlessly fit into existing DM pipelines. Experimental results demonstrate that Naïve PAINE outperforms existing approaches on several prompt corpus benchmarks.

Naïve PAINE: Lightweight Text-to-Image Generation Improvement with Prompt Evaluation

Abstract

Text-to-Image (T2I) generation is primarily driven by Diffusion Models (DM) which rely on random Gaussian noise. Thus, like playing the slots at a casino, a DM will produce different results given the same user-defined inputs. This imposes a gambler's burden: To perform multiple generation cycles to obtain a satisfactory result. However, even though DMs use stochastic sampling to seed generation, the distribution of generated content quality highly depends on the prompt and the generative ability of a DM with respect to it. To account for this, we propose Naïve PAINE for improving the generative quality of Diffusion Models by leveraging T2I preference benchmarks. We directly predict the numerical quality of an image from the initial noise and given prompt. Naïve PAINE then selects a handful of quality noises and forwards them to the DM for generation. Further, Naïve PAINE provides feedback on the DM generative quality given the prompt and is lightweight enough to seamlessly fit into existing DM pipelines. Experimental results demonstrate that Naïve PAINE outperforms existing approaches on several prompt corpus benchmarks.
Paper Structure (25 sections, 5 equations, 10 figures, 10 tables)

This paper contains 25 sections, 5 equations, 10 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Noise-Aware Optimization
  4. Human Preference Metrics
  5. Human Preferences and Prompt Influence
  6. Methodology
  7. Prompt Evaluation Prior to Generation
  8. Naïve PAINE: Prompt Generation Performance Estimation
  9. Results
  10. Predictor Architecture, Datasets, Training and Evaluation
  11. T2I Evaluation
  12. Prompt-Aware DM Capability Estimation
  13. Ablation Studies
  14. Conclusions and Future Work
  15. Additional Implementation Details
  16. ...and 10 more sections

Figures (10)

  • Figure 1: Measuring the PickScore kirstain2023pick distribution across several prompts and DMs podell2023sdxllykon2023dreamshaperli2024hunyuanditchen2024pixartsigma. Specifically, we select the first 50 training prompts from Golden Noise zhou2025golden and generate 20 images per prompt, resetting the random seed prior to the first generation. Then, we plot the PickScore score mean and standard deviation (error bar).
  • Figure 2: Pearson Correlation Coefficient (PCC) matrix comparing the mean PickScore for each DM. 'D.S.' and 'H.Y.' shorthand for lykon2023dreamshaper and li2024hunyuandit.
  • Figure 3: Pearson Correlation Coefficient (PCC) matrix comparing the mean PickScore across prompts for lykon2023dreamshaper.
  • Figure 4: We augment a DM pipeline with Naïve PAINE which receives the prompt embedding $c$ and $N$ noises as input to estimate the human preference score metric as if each noise were processed into an image. We sort and rank the scores of the $N$ noises and forward the top-$|B|$ noises to the DM for normal generation. Further, score estimations provide feedback to the user on how well the DM can perform on a given prompt. Image from Hunyuan. Scores from our dataset and method on PickScore kirstain2023pick.
  • Figure 5: Qualitative visual examples. DM and prompt details provided. Best viewed in color. We provide further examples in the supplementary materials.
  • ...and 5 more figures