Your Latent Mask is Wrong: Pixel-Equivalent Latent Compositing for Diffusion Models

TL;DR

<3-5 sentence high-level summary>The paper identifies a fundamental mismatch between pixel-space masking and latent-space compositing in modern diffusion models due to nonlinear VAEs with wide receptive fields, which causes artifacts at mask boundaries and global color shifts. It introduces Pixel-Equivalent Latent Compositing (PELC), a decoder- and encoder-equivalence framework, and presents DecFormer, a lightweight transformer that learns per-channel blend weights and a residual to achieve pixel-consistent latent fusion. Trained with pixel-space supervision, PELC operates as a drop-in replacement for latent blending with minimal overhead and is plug-compatible with existing diffusion pipelines. Experiments show substantial improvements in boundary fidelity and soft-mask support, and a small LoRA extension can match the quality of fully finetuned inpainting models, illustrating practical impact for high-resolution, editing-oriented diffusion workflows.

Abstract

Latent inpainting in diffusion models still relies almost universally on linearly interpolating VAE latents under a downsampled mask. We propose a key principle for compositing image latents: Pixel-Equivalent Latent Compositing (PELC). An equivalent latent compositor should be the same as compositing in pixel space. This principle enables full-resolution mask control and true soft-edge alpha compositing, even though VAEs compress images 8x spatially. Modern VAEs capture global context beyond patch-aligned local structure, so linear latent blending cannot be pixel-equivalent: it produces large artifacts at mask seams and global degradation and color shifts. We introduce DecFormer, a 7.7M-parameter transformer that predicts per-channel blend weights and an off-manifold residual correction to realize mask-consistent latent fusion. DecFormer is trained so that decoding after fusion matches pixel-space alpha compositing, is plug-compatible with existing diffusion pipelines, requires no backbone finetuning and adds only 0.07% of FLUX.1-Dev's parameters and 3.5% FLOP overhead. On the FLUX.1 family, DecFormer restores global color consistency, soft-mask support, sharp boundaries, and high-fidelity masking, reducing error metrics around edges by up to 53% over standard mask interpolation. Used as an inpainting prior, a lightweight LoRA on FLUX.1-Dev with DecFormer achieves fidelity comparable to FLUX.1-Fill, a fully finetuned inpainting model. While we focus on inpainting, PELC is a general recipe for pixel-equivalent latent editing, as we demonstrate on a complex color-correction task.

Figures (10)

• Figure 1: Each quadrant compares ground-truth pixel composites, our DecFormer predictions, and heuristic latent interpolation. Across soft, binary, and structured masks, DecFormer restores sharp edges and high-frequency detail, whereas the heuristic exhibits smearing and artifacts on soft blends, halos and discoloration at boundaries, and blocky low-fidelity masks. Notably, in the bottom-right example, global background degradation occurs far from the masked region, reflecting how latent entanglement corrupts off-mask content; this effect is eliminated by DecFormer.
• Figure 2: Effective Receptive Field (ERF) analysis of the Flux VAE on $256{\times}256$ images; panels (a–c) are randomly chosen near the pooled median. Statistics pool $128{\times}3$ probes with 95% bootstrap CIs; radii $r_{50},r_{90}$ are cumulative-energy radii (fraction of image/latent diagonal). (a) Decoder FD ERF: perturb one latent site by $\varepsilon$ in all channels and plot $\|D(z+\varepsilon e)-D(z)\|_2$ per pixel (log). Visualization uses adaptive $\varepsilon$ ($\sim$0.5% decoded RMS; clamp $[2{\times}10^{-3},10^{-2}]$); metrics use fixed $\varepsilon{=}10^{-3}$. Radii: $r_{50}\!\approx\!0.044\pm0.024$, $r_{90}\!\approx\!0.291\pm0.099$. The bright core plus global low-amplitude ‘blanket’ highlights decoder non-locality, seems to echo high contrast structures in source image. (b) Encoder FD ERF: inject a 1px impulse ($\delta{=}0.05$) at pixel $(i,j)$, compute $\Delta z{=}E(x+\delta e_{i,j}){-}E(x)$, and plot $\|\Delta z_{\cdot,u,v}\|_2$ per latent site (log). Radii: $r_{50}\!\approx\!0.091\pm0.082$, $r_{90}\!\approx\!0.356\pm0.099$. (c) Gradient ERF: for $y{=}D(z)$ and $s{=}\sum_{\text{5}\times\text{5}} y^2$, backpropagate to $\partial s/\partial z$ and show channelwise $\ell_2$ per latent site. Central peak and multiple secondary latent clusters relied on by the same pixel patch, exposing repeated structure. (d) Energy curves: Shaded region shows 95% bootstrap confidence; both Encoder and Decoder shows a sharp core with long, low-amplitude tails, showing large ERFs; evidence that heuristic latent masking is inconsistent with the VAE and motivates DecFormer.
• Figure 3: Overview of our training pipeline and DecFormer architecture. The left panel illustrates the overall flow: two input images and a pixel mask are encoded by a frozen VAE, the mask is processed by a lightweight CNN prior (architecture detailed in Appendix \ref{['blind-pred-arch']}), and DecFormer predicts channel-adaptive blend weights $\alpha$ and residual corrections $s$ at latent resolution. The right panel zooms into a single DecFormer block, showing the feature stack, patching/unpatching, FiLM conditioning, attention and cross attention. For an expanded diagram on the DecFormer architecture see Appendix \ref{['deltaformer-extended-arch']}.
• Figure 4: Signed-distance analysis of mask edges. Left: example masks and their signed distance fields (SDF). Middle: Mean per-latent MSE relative to latent downsized mask SDF. Right: Mean per-pixel MSE relative to mask SDF. Heuristic (grey) applies the downsampled binary mask in latent space; Alpha‑clamped (orange) uses the clamped least-squares solved $\alpha^*$ Appendix \ref{['app:projection']}; DecFormer (green) is our model. DecFormer achieves the lowest error around the boundary with a sharper fall-off compared to baselines.
• Figure 5: Inpainting quality comparisons for 4 images between heuristic, DecFormer, light-training inpainting LoRA, LoRA with DecFormer, and Flux.1 Fill, a fully finetuned image editing model.