Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Consistency-Based Image Generation via Adversarialy-Trained Classification and Energy-Based Discrimination

Shelly Golan, Roy Ganz, Michael Elad

TL;DR

This paper tackles the fidelity gap of fast-consistency image generation by introducing a post-processing technique that trains a joint classifier–discriminator adversarially to provide perceptually aligned gradients for refinement. By combining a robust classification objective with energy-based discrimination, and applying example-specific projected gradient steps with early stopping, the method boosts image realism for Consistency-Training and Consistency-Distillation across ImageNet 64×64 and other resolutions. The approach yields substantial improvements in Fréchet Inception Distance and related metrics, outperforming prior baselines such as BIGROC and ACT while offering favorable runtime trade-offs. The results suggest a general, model-agnostic mechanism to enhance fidelity in fast generative pipelines through joint adversarial guidance grounded in energy-based reasoning.

Abstract

The recently introduced Consistency models pose an efficient alternative to diffusion algorithms, enabling rapid and good quality image synthesis. These methods overcome the slowness of diffusion models by directly mapping noise to data, while maintaining a (relatively) simpler training. Consistency models enable a fast one- or few-step generation, but they typically fall somewhat short in sample quality when compared to their diffusion origins. In this work we propose a novel and highly effective technique for post-processing Consistency-based generated images, enhancing their perceptual quality. Our approach utilizes a joint classifier-discriminator model, in which both portions are trained adversarially. While the classifier aims to grade an image based on its assignment to a designated class, the discriminator portion of the very same network leverages the softmax values to assess the proximity of the input image to the targeted data manifold, thereby serving as an Energy-based Model. By employing example-specific projected gradient iterations under the guidance of this joint machine, we refine synthesized images and achieve an improved FID scores on the ImageNet 64x64 dataset for both Consistency-Training and Consistency-Distillation techniques.

Enhancing Consistency-Based Image Generation via Adversarialy-Trained Classification and Energy-Based Discrimination

TL;DR

This paper tackles the fidelity gap of fast-consistency image generation by introducing a post-processing technique that trains a joint classifier–discriminator adversarially to provide perceptually aligned gradients for refinement. By combining a robust classification objective with energy-based discrimination, and applying example-specific projected gradient steps with early stopping, the method boosts image realism for Consistency-Training and Consistency-Distillation across ImageNet 64×64 and other resolutions. The approach yields substantial improvements in Fréchet Inception Distance and related metrics, outperforming prior baselines such as BIGROC and ACT while offering favorable runtime trade-offs. The results suggest a general, model-agnostic mechanism to enhance fidelity in fast generative pipelines through joint adversarial guidance grounded in energy-based reasoning.

Abstract

The recently introduced Consistency models pose an efficient alternative to diffusion algorithms, enabling rapid and good quality image synthesis. These methods overcome the slowness of diffusion models by directly mapping noise to data, while maintaining a (relatively) simpler training. Consistency models enable a fast one- or few-step generation, but they typically fall somewhat short in sample quality when compared to their diffusion origins. In this work we propose a novel and highly effective technique for post-processing Consistency-based generated images, enhancing their perceptual quality. Our approach utilizes a joint classifier-discriminator model, in which both portions are trained adversarially. While the classifier aims to grade an image based on its assignment to a designated class, the discriminator portion of the very same network leverages the softmax values to assess the proximity of the input image to the targeted data manifold, thereby serving as an Energy-based Model. By employing example-specific projected gradient iterations under the guidance of this joint machine, we refine synthesized images and achieve an improved FID scores on the ImageNet 64x64 dataset for both Consistency-Training and Consistency-Distillation techniques.
Paper Structure (19 sections, 19 equations, 4 figures, 5 tables, 3 algorithms)

This paper contains 19 sections, 19 equations, 4 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Diffusion Models and Consistency
  4. Boosting Synthesis via Robust Classifier (BIGROC)
  5. Energy-Based Models
  6. Proposed Approach
  7. Training a Joint Classifier-Discriminator
  8. Proposed Training objective
  9. Gradual Adversarial Training
  10. Sampling
  11. Experiments
  12. Experimental Settings
  13. Experimental Results
  14. Ablation Study
  15. Conclusion
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Top: Images generated by Consistency Models trained on ImageNet 64x64. Bottom: Refined images by applying our algorithm.
  • Figure 2: Top: Refined images by applying our algorithm. Bottom: Images generated by Guided Diffusion trained on ImageNet 256x256.
  • Figure 3: Top: Images generated by Guided Diffusion trained on ImageNet 256x256. Bottom: Refined images by applying our algorithm.
  • Figure 4: Top: Images generated by Guided Diffusion trained on ImageNet 128x128. Bottom: Refined images by applying our algorithm.