Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human Expression

TL;DR

Style Tailoring is introduced, a recipe to finetune Latent Diffusion Models in a distinct domain with high visual quality, prompt alignment and scene diversity, and a novel fine-tuning method called Style Tailoring, which jointly fits the content and style distribution and achieves best tradeoff.

Abstract

We introduce Style Tailoring, a recipe to finetune Latent Diffusion Models (LDMs) in a distinct domain with high visual quality, prompt alignment and scene diversity. We choose sticker image generation as the target domain, as the images significantly differ from photorealistic samples typically generated by large-scale LDMs. We start with a competent text-to-image model, like Emu, and show that relying on prompt engineering with a photorealistic model to generate stickers leads to poor prompt alignment and scene diversity. To overcome these drawbacks, we first finetune Emu on millions of sticker-like images collected using weak supervision to elicit diversity. Next, we curate human-in-the-loop (HITL) Alignment and Style datasets from model generations, and finetune to improve prompt alignment and style alignment respectively. Sequential finetuning on these datasets poses a tradeoff between better style alignment and prompt alignment gains. To address this tradeoff, we propose a novel fine-tuning method called Style Tailoring, which jointly fits the content and style distribution and achieves best tradeoff. Evaluation results show our method improves visual quality by 14%, prompt alignment by 16.2% and scene diversity by 15.3%, compared to prompt engineering the base Emu model for stickers generation.

Figures (10)

• Figure 1: Stickers generated by our text-to-sticker model. They are visually pleasing, diverse, and with high text faithfulness.
• Figure 2: Architecture of our text-to-sticker model (left) and transparency decoder (right). The text-to-sticker model contains an optional prompt enhancer, a diffusion U-Net, and frozen text conditioning added via cross-attention. The transparency decoder is augemented from a regular VAE decoder, by adding one extra output channel at final output conv layer. The alpha-channel convolution weights are initialized with the average of R, G, B channels' weights.
• Figure 3: Illustration of our text-to-sticker model finetuning recipe. (a) Standard multi-stage fine-tuning. (b) Our proposed method, Style Tailoring. In Style Tailoring, we implement a phased dataloader such that the U-Net denoising steps $T$ to $T'+1$ are trained with HITL alignment data (content distribution $p_{content}$), and denoising steps $T'$ to $0$ are trained with EITL data (style distribution $p_{style}$).
• Figure 4: Qualitative results for the models in Table \ref{['table:method_comparison']}. Baseline (Row 1) lacks prompt alignment and diversity, domain aligned model (Row 2) improves alignment and diversity but is much worse in quality. Multi-stage finetuning (Rows 3 & 4) face a trade off between prompt and style alignment. Style Tailoring (Row 5) offers the best results in both prompt and style alignment. More qualitative examples in the supplementary.
• Figure 5: Generalization of Style tailoring to multiple styles: our final, target graphic style (top row) and alternate volumetric style (bottom row).