Generation is Required for Data-Efficient Perception

TL;DR

The paper questions whether human-like, data-efficient visual perception requires a generative approach. It develops a formal theory comparing generative (decoder-inversion) and non-generative (encoder) methods under a compositional data-generating process, proving that encoders generally cannot be constrained to the necessary biases for OOD generalization, while decoders can be constrained to a class that guarantees it. Empirically, non-generative models struggle on OOD tasks without large-scale pretraining or supervision, whereas generative models achieve substantial compositional generalization by leveraging decoder inductive biases and inference techniques like gradient-based search and generative replay. Across photorealistic PUG datasets, the generative approach yields significant OOD gains without extra data, highlighting a principled pathway toward data-efficient perception via decoder-based inversion. The work suggests future directions for scaling generative-inversion methods and developing rigorous benchmarks for compositional generalization in realistic settings.

Abstract

It has been hypothesized that human-level visual perception requires a generative approach in which internal representations result from inverting a decoder. Yet today's most successful vision models are non-generative, relying on an encoder that maps images to representations without decoder inversion. This raises the question of whether generation is, in fact, necessary for machines to achieve human-level visual perception. To address this, we study whether generative and non-generative methods can achieve compositional generalization, a hallmark of human perception. Under a compositional data generating process, we formalize the inductive biases required to guarantee compositional generalization in decoder-based (generative) and encoder-based (non-generative) methods. We then show theoretically that enforcing these inductive biases on encoders is generally infeasible using regularization or architectural constraints. In contrast, for generative methods, the inductive biases can be enforced straightforwardly, thereby enabling compositional generalization by constraining a decoder and inverting it. We highlight how this inversion can be performed efficiently, either online through gradient-based search or offline through generative replay. We examine the empirical implications of our theory by training a range of generative and non-generative methods on photorealistic image datasets. We find that, without the necessary inductive biases, non-generative methods often fail to generalize compositionally and require large-scale pretraining or added supervision to improve generalization. By comparison, generative methods yield significant improvements in compositional generalization, without requiring additional data, by leveraging suitable inductive biases on a decoder along with search and replay.

Figures (7)

• Figure 1: Generative vs. non-generative compositional generalization. We assume in-domain (ID) and out-of-domain (OOD) images arise from a latent variable model through an unknown compositional generator $\bm{f} \in \mathcal{F}_\textnormal{int}$, with inverse $\bm{g} \in \mathcal{G}_\textnormal{int}$. Guaranteeing compositional generalization for a generative approach requires constraining a decoder such that $\hat{\bm{f}} \in \mathcal{F}_\textnormal{int}$, and for a non-generative approach, an encoder such that $\hat{\bm{g}} \in \mathcal{G}_\textnormal{int}$ (Sec. \ref{['sec:setup']}). We show theoretically in (Sec. \ref{['sec:theory']}) that placing such constraints on an encoder is generally infeasible with inductive biases while for a decoder it is straightforward. Empirically, this tends to manifest in an encoder yielding incorrect representations for OOD images (Sec. \ref{['sec:empir_res']}). In contrast, a decoder is able to correctly generate such images enabling compositional generalization through inversion (Sec. \ref{['sec:search_replay']}, \ref{['sec:empir_res_gen']}).
• Figure 2: Visualization of a data generating process with in- and out-of-domain regions.
• Figure 3: Structure of a data manifold $\mathcal{X}$ and latent manifold $\mathcal{Z}$.
• Figure 4: Approaches for inverting a generator out-of-domain.Left. Visualization of gradient-based search to invert a decoder $\hat{\bm{f}}$ out-of-domain, with initialization given by an encoder $\hat{\bm{g}}$. Right. Visualization of generative replay in which an encoder is trained on OOD images generated by a decoder.
• Figure 5: ID and OOD images from each dataset used in our experiments.

Theorems & Definitions (15)

• Lemma 3.0
• Theorem 3.1
• Definition A.1: Additive functions
• Lemma A.1
• Remark A.2
• Lemma A.3
• Lemma A.4
• Lemma A.5
• Remark A.6
• Lemma A.7