Learning to Generate Chairs, Tables and Cars with Convolutional Networks

TL;DR

The paper presents up-convolutional, conditioned generative networks that synthesize high-resolution images of chairs, tables, and cars from high-level descriptors (style, viewpoint, and transformations). By combining a learned latent space with an expanding upsampling path, the method enables view interpolation, elevation transfer, style morphing, and cross-class knowledge transfer, demonstrating that the networks capture meaningful 3D structure rather than memorizing samples. A probabilistic extension with a latent z and variational training enables principled sampling of novel objects. The work also analyzes internal representations, showing transformation-specific neurons and meaningful interactions that produce sharp, controllable images, and demonstrates practical applications such as dense correspondences between different object instances. Overall, the approach advances supervised generative modeling for large, conditioned image generation and cross-object 3D understanding, with potential to scale to more object classes.

Abstract

We train generative 'up-convolutional' neural networks which are able to generate images of objects given object style, viewpoint, and color. We train the networks on rendered 3D models of chairs, tables, and cars. Our experiments show that the networks do not merely learn all images by heart, but rather find a meaningful representation of 3D models allowing them to assess the similarity of different models, interpolate between given views to generate the missing ones, extrapolate views, and invent new objects not present in the training set by recombining training instances, or even two different object classes. Moreover, we show that such generative networks can be used to find correspondences between different objects from the dataset, outperforming existing approaches on this task.

Figures (25)

• Figure 1: Architecture of a 1-stream deep network ("1s-S-deep") that generates $128 \times 128$ pixel images. Layer names are shown above: FC - fully connected, upconv - upsampling+convolution, conv - convolution.
• Figure 2: Illustration of upsampling (left) and upsampling+convolution (right) as used in the generative network.
• Figure 3: Representative images used for training the networks. Top: chairs, middle: tables, bottom: cars.
• Figure 4: Qualitative results with different networks trained on chairs. See the description of architectures in section \ref{['sec:arch_comparison']}.
• Figure 5: Qualitative results for different numbers of car models in the training set, without and with (rightmost column) data augmentation.