Scaling Non-Parametric Sampling with Representation

TL;DR

This work proposes a minimal, white-box non-parametric image generator that samples each pixel from an empirical distribution derived from similar context patches, conditioned on a compact representation. It integrates three principles—spatial non-stationarity, low-level statistics, and high-level semantics—through a shell-based image synthesis workflow and three distance metrics, including a self-supervised embedding for semantic matching. The model, despite no training, produces high-fidelity MNIST and visually compelling CIFAR-10 samples and enables mechanistic interpretation via source-tracing, revealing a part–whole generalization mechanism where coherent regions are assembled from multiple sources while preserving class-level coherence. These findings offer a concrete hypothesis for how complex neural generators might learn to generalize through simple, compositional rules, and suggest directions for simpler, interpretable future models that bridge the gap between white-box reasoning and black-box performance.

Abstract

Scaling and architectural advances have produced strikingly photorealistic image generative models, yet their mechanisms still remain opaque. Rather than advancing scaling, our goal is to strip away complicated engineering tricks and propose a simple, non-parametric generative model. Our design is grounded in three principles of natural images-(i) spatial non-stationarity, (ii) low-level regularities, and (iii) high-level semantics-and defines each pixel's distribution from its local context window. Despite its minimal architecture and no training, the model produces high-fidelity samples on MNIST and visually compelling CIFAR-10 images. This combination of simplicity and strong empirical performance points toward a minimal theory of natural-image structure. The model's white-box nature also allows us to have a mechanistic understanding of how the model generalizes and generates diverse images. We study it by tracing each generated pixel back to its source images. These analyses reveal a simple, compositional procedure for "part-whole generalization", suggesting a hypothesis for how large neural network generative models learn to generalize.

Figures (12)

• Figure 1: Non-Parametric Sampling Conditioned on Representation. The representation model computes some latent encoding before the last layer, and we extract that model with early exit, allowing L2-norm comparison of the resulting latent representations.
• Figure 2: Ablation Study Comparison of Statistical Components. Each image shows the improvement from progressively adding our principle components to the generator.
• Figure 3: Samples from Non-Parametric Image Generation. We show the generation result for non-parametric with non-stationary, low-level and high-level statistics: MNIST (left), and CIFAR-10 (right).
• Figure 4: MNIST Samples from Class-Conditional Non-Parametric Model. Samples here are generated from a model with non-stationary and low-level statistics as described in Section \ref{['N,L']}, together with class label conditioning.
• Figure 5: Viewing Overlapping Inter-Class Features. (a) This sample does not resemble a digit. b) The corresponding class source-map reveals conflict between classes 3, 5, and 6. c) This is another unique sample, this time with CIFAR-10. d) It's corresponding source map shows an even more interesting scenario, where two images of ship and plane are combined due to the shared sky in the background.