BiGR: Harnessing Binary Latent Codes for Image Generation and Improved Visual Representation Capabilities

TL;DR

BiGR addresses the dual goals of high-quality conditional image generation and strong visual representations. It introduces a binary latent-code pipeline with a binary tokenizer, a bidirectional transformer, and a binary transcoder, trained via masked modeling and denoised through a Bernoulli diffusion process. An entropy-guided sampling scheme enables efficient generation, while average-pooled mid-layer features yield robust linear-probe representations, achieving competitive generation metrics and superior discriminative capabilities. The approach supports zero-shot vision tasks (inpainting, outpainting, editing, interpolation, enrichment) and extends to text-to-image generation with minimal adaptations, illustrating a unified framework that bridges generative and discriminative objectives with practical, scalable performance.

Abstract

We introduce BiGR, a novel conditional image generation model using compact binary latent codes for generative training, focusing on enhancing both generation and representation capabilities. BiGR is the first conditional generative model that unifies generation and discrimination within the same framework. BiGR features a binary tokenizer, a masked modeling mechanism, and a binary transcoder for binary code prediction. Additionally, we introduce a novel entropy-ordered sampling method to enable efficient image generation. Extensive experiments validate BiGR's superior performance in generation quality, as measured by FID-50k, and representation capabilities, as evidenced by linear-probe accuracy. Moreover, BiGR showcases zero-shot generalization across various vision tasks, enabling applications such as image inpainting, outpainting, editing, interpolation, and enrichment, without the need for structural modifications. Our findings suggest that BiGR unifies generative and discriminative tasks effectively, paving the way for further advancements in the field. We further enable BiGR to perform text-to-image generation, showcasing its potential for broader applications.

Figures (23)

• Figure 1: BiGR generates high-quality images while improving the discriminative capabilities of the representations.Left: Generated 512$\times$512 samples, 256$\times$256 samples, and class-conditional editing samples. Right: BiGR vs. LlamaGen sun2024autoregressive. We visualize image features from 100 classes in ImageNet-1K validation split using t-SNE tsne, where the same color indicates the same class. Our model produces features with greater discriminative separability and enhances both generative and discriminative performance.
• Figure 2: Overview of BiGR. For simplicity, we display only 1 bit for each token, although each token actually consists of K bits in length. Left: We outline the training of BiGR. Starting with binary codes from binary tokenizers, we append a condition token and mask partial tokens. These tokens are projected into continuous embeddings and processed by the Llama backbone. The outputs undergo a Bernoulli denoising process in the binary transcoder to generate probabilities, penalized by the weighted binary cross-entropy loss (wBCE) at masked positions. Right: We illustrate the generation process (detailed in \ref{['sec:sampling']}) and the representation acquisition via average pooling.
• Figure 3: Relationships between FID-50K and sample time across varying inference hyperparameters. We compare different numbers of sampling iterations $N$ (left) and varying diffusion timesteps $T$ (right) for three model sizes. All other hyperparameters are kept at their default settings.
• Figure 4: Evaluation of generative and discriminative performance across different model sizes. We report results for all tested tokenizers across four different dimensions of binary codes. We include the reconstruction FID (rFID) for each binary tokenizer for reference (grey points).
• Figure 5: Generated 512$\times$512 samples.