NOFT: Test-Time Noise Finetune via Information Bottleneck for Highly Correlated Asset Creation

TL;DR

NOFT tackles the challenge of generating highly correlated yet diverse 2D/3D assets by finetuning the noise latent at test time rather than modifying diffusion priors or relying on external control signals. It introduces an Optimal-Transported Information Bottleneck (OTIB) framework with a Sinkhorn attention module to simultaneously preserve topology/texture and promote diversity, while keeping the diffusion model frozen. The method achieves efficient plug-and-play operation with around 14K trainable parameters and short training times, demonstrating competitive or superior fidelity and variety compared to state-of-the-art structure- and texture-guided baselines. This approach enables robust, controllable asset creation under text or image guidance with implications for design workflows, game development, and digital content creation, albeit with considerations for misuse and the need for watermarking.

Abstract

The diffusion model has provided a strong tool for implementing text-to-image (T2I) and image-to-image (I2I) generation. Recently, topology and texture control are popular explorations, e.g., ControlNet, IP-Adapter, Ctrl-X, and DSG. These methods explicitly consider high-fidelity controllable editing based on external signals or diffusion feature manipulations. As for diversity, they directly choose different noise latents. However, the diffused noise is capable of implicitly representing the topological and textural manifold of the corresponding image. Moreover, it's an effective workbench to conduct the trade-off between content preservation and controllable variations. Previous T2I and I2I diffusion works do not explore the information within the compressed contextual latent. In this paper, we first propose a plug-and-play noise finetune NOFT module employed by Stable Diffusion to generate highly correlated and diverse images. We fine-tune seed noise or inverse noise through an optimal-transported (OT) information bottleneck (IB) with around only 14K trainable parameters and 10 minutes of training. Our test-time NOFT is good at producing high-fidelity image variations considering topology and texture alignments. Comprehensive experiments demonstrate that NOFT is a powerful general reimagine approach to efficiently fine-tune the 2D/3D AIGC assets with text or image guidance.

Figures (19)

• Figure 1: Our method noise finetune (NOFT) completely decouples highly correlated manifold representation learning from dependencies of concept images worddreambooth and external control signals controlnetuni, as well as pre-trained T2I model finetuning ipwordcontrolnet. Test-time NOFT facilitates high-quality 2D assets latentdiffsd3 and 3D assets trellis with high contextual fidelity and controllable diversity, under any text or image condition (denoted by red boxes). Zoom in for better observation or go to the Appendix.
• Figure 2: Content-diversity tradeoff: given two distinguished noises, we obtain $N_{NOFT}$ by finetuning $N_{1}$ where adaptively injecting $N_{2}$ based on information bottleneck. The corresponding denoised images are $I_{1}, I_{2}, I_{NOFT}$. As shown by $I_{NOFT}$, the structure and appearance statistics from $I_{1}$ are preserved well, with concurrently improved diversity inherited from the local topological statistic from $I_{2}$.
• Figure 3: Feature workbench provided by DSG self is fine-grained but cumbersome. Our NOFT gives another efficient and diverse workbench to change the properties of objects.
• Figure 4: Method overview: as a plug-and-play content controller, NOFT can be employed for 2D/3D generation tasks, different architectures and model checkpoints. NOFT consists of a Sinkhorn Attention module and an information bottleneck module. We obtain $N_{NOFT}$ by information compression of $N_{Orig}$ and information modulation of $N_{Div}$. More details are introduced in Section \ref{['Approach']}.
• Figure 5: Qualitative results of NOFT_2D, ControlNet + IP Adapter controlnetip, FreeControl freecontrol, Ctrl-X ctrlx, Uni-ControlNet uni, T2I-Adapter + IP Adapter t2i-adaptip, and Reimagine reimagine. Zoom in for better observation. NOFT realizes more controllable image variations with high-fidelity content.