Bag of Design Choices for Inference of High-Resolution Masked Generative Transformer

TL;DR

This work proposes and redesigns a set of enhanced inference techniques tailored for MGT, providing a detailed analysis of their performance and explores several DM-based approaches aimed at accelerating the sampling process on MGT.

Abstract

Text-to-image diffusion models (DMs) develop at an unprecedented pace, supported by thorough theoretical exploration and empirical analysis. Unfortunately, the discrepancy between DMs and autoregressive models (ARMs) complicates the path toward achieving the goal of unified vision and language generation. Recently, the masked generative Transformer (MGT) serves as a promising intermediary between DM and ARM by predicting randomly masked image tokens (i.e., masked image modeling), combining the efficiency of DM with the discrete token nature of ARM. However, we find that the comprehensive analyses regarding the inference for MGT are virtually non-existent, and thus we aim to present positive design choices to fill this gap. We propose and redesign a set of enhanced inference techniques tailored for MGT, providing a detailed analysis of their performance. Additionally, we explore several DM-based approaches aimed at accelerating the sampling process on MGT. Extensive experiments and empirical analyses on the recent SOTA MGT, such as MaskGIT and Meissonic lead to concrete and effective design choices, and these design choices can be merged to achieve further performance gains. For instance, in terms of enhanced inference, we achieve winning rates of approximately 70% compared to vanilla sampling on HPS v2 with Meissonic-1024x1024.

Figures (15)

• Figure 1: Visualization of our design choices on Meissonic-512$\times$512 and Meissonic-1024$\times$1024. Noise regularization, differential sampling, and Z-Sampling significantly improve both visual quality and semantic faithfulness. Additionally, our quantization method secondary calibration quantization (SCQ) can reduce the memory footprint without significant performance degradation.
• Figure 2: The complete sampling pipeline of MaskGIT and Meissonic, and how our proposed effective and specific design choices integrate into that sampling pipeline. Specifically, TomeMGT and SCQ act on the Transformer to reduce inference latency and memory usage, respectively. Meanwhile, noise regularization and differential sampling enhance inference by correcting the probability distribution applied for sampling. Additionally, masked Z-Sampling is a rescheduling technique that significantly improves the quality of synthesized images through the forward-inversion operator (i.e., sampling and backtracking alternatively).
• Figure 3: Visualization of different noise schedules. The black dashed line represents the cosine schedule.
• Figure 4: Visualization of the performance of different noise schedules. The dotted line denotes the vanilla sampling.
• Figure 5: The illustration of vanilla Z-Sampling and masked Z-Sampling. The main difference is the masking form.