Efficient inference in occlusion-aware generative models of images

TL;DR

This work tackles occlusion and data-label bottlenecks in image understanding by introducing CST-VAE, a probabilistic, layered model that composes multiple object layers from front to back. Each layer is generated by a ST-VAE that disentangles content from pose and is warped via Spatial Transformer Networks, enabling unsupervised, interpretable layer decomposition. Trained with stochastic gradient variational Bayes, CST-VAE learns latent representations that separate foreground and background and improve downstream classification on occluded scenes. Empirical results on MNIST-based datasets show improved likelihoods and clear foreground/background separation, highlighting a path toward unsupervised, occlusion-aware scene understanding.

Abstract

We present a generative model of images based on layering, in which image layers are individually generated, then composited from front to back. We are thus able to factor the appearance of an image into the appearance of individual objects within the image --- and additionally for each individual object, we can factor content from pose. Unlike prior work on layered models, we learn a shape prior for each object/layer, allowing the model to tease out which object is in front by looking for a consistent shape, without needing access to motion cues or any labeled data. We show that ordinary stochastic gradient variational bayes (SGVB), which optimizes our fully differentiable lower-bound on the log-likelihood, is sufficient to learn an interpretable representation of images. Finally we present experiments demonstrating the effectiveness of the model for inferring foreground and background objects in images.

Figures (7)

• Figure 1: Illustration of occlusion.
• Figure 2: \ref{['fig:cartoon']} Cartoon illustration of the CST-VAE layer compositing process; \ref{['fig:compositestvae_gm']} CST-VAE graphical model.
• Figure 3: \ref{['fig:stvae_decoder']} ST-VAE Generative model, $P(L | z^C, z^T)$ (Decoder); \ref{['fig:stvae_encoder']} ST-VAE Recognition model $Q(z^C,z^T | L) = Q(z^C | z^T, L)\cdot Q(z^T | L)$ (Encoder)
• Figure 4: The CST-VAE network "unrolled" for two image layers.
• Figure 5: \ref{['fig:vae_stvae_samples']} Comparison of samples from the VAE and ST-VAE generative models. For the ST-VAE model, we show both the sample in its canonical pose and the final generated image. \ref{['fig:stvae_averageddigits']} Averaged images from each MNIST class as learning progresses --- we typically see pose variables converge very quickly.