Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentation

You Wu, Kean Liu, Xiaoyue Mi, Fan Tang, Juan Cao, Jintao Li

TL;DR

U-VAP tackles fine-grained, user-guided visual appearance personalization for diffusion models by introducing a decoupled self-augmentation framework that separately learns target and non-target attribute embeddings. It leverages LLM-driven prompt generation to create attribute-aware augmentations, uses data curation to form dual-concept learning sets, and applies semantic adjustment during inference to further disentangle attributes. The approach yields improved controllability and flexibility over state-of-the-art personalization methods across color, pattern, and structure attributes, demonstrated through quantitative metrics and qualitative comparisons. This work advances practical personalization by enabling precise attribute control and compositional concept generation while acknowledging reliance on base personalization models and potential language-prior limitations.

Abstract

Concept personalization methods enable large text-to-image models to learn specific subjects (e.g., objects/poses/3D models) and synthesize renditions in new contexts. Given that the image references are highly biased towards visual attributes, state-of-the-art personalization models tend to overfit the whole subject and cannot disentangle visual characteristics in pixel space. In this study, we proposed a more challenging setting, namely fine-grained visual appearance personalization. Different from existing methods, we allow users to provide a sentence describing the desired attributes. A novel decoupled self-augmentation strategy is proposed to generate target-related and non-target samples to learn user-specified visual attributes. These augmented data allow for refining the model's understanding of the target attribute while mitigating the impact of unrelated attributes. At the inference stage, adjustments are conducted on semantic space through the learned target and non-target embeddings to further enhance the disentanglement of target attributes. Extensive experiments on various kinds of visual attributes with SOTA personalization methods show the ability of the proposed method to mimic target visual appearance in novel contexts, thus improving the controllability and flexibility of personalization.

U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentation

TL;DR

U-VAP tackles fine-grained, user-guided visual appearance personalization for diffusion models by introducing a decoupled self-augmentation framework that separately learns target and non-target attribute embeddings. It leverages LLM-driven prompt generation to create attribute-aware augmentations, uses data curation to form dual-concept learning sets, and applies semantic adjustment during inference to further disentangle attributes. The approach yields improved controllability and flexibility over state-of-the-art personalization methods across color, pattern, and structure attributes, demonstrated through quantitative metrics and qualitative comparisons. This work advances practical personalization by enabling precise attribute control and compositional concept generation while acknowledging reliance on base personalization models and potential language-prior limitations.

Abstract

Concept personalization methods enable large text-to-image models to learn specific subjects (e.g., objects/poses/3D models) and synthesize renditions in new contexts. Given that the image references are highly biased towards visual attributes, state-of-the-art personalization models tend to overfit the whole subject and cannot disentangle visual characteristics in pixel space. In this study, we proposed a more challenging setting, namely fine-grained visual appearance personalization. Different from existing methods, we allow users to provide a sentence describing the desired attributes. A novel decoupled self-augmentation strategy is proposed to generate target-related and non-target samples to learn user-specified visual attributes. These augmented data allow for refining the model's understanding of the target attribute while mitigating the impact of unrelated attributes. At the inference stage, adjustments are conducted on semantic space through the learned target and non-target embeddings to further enhance the disentanglement of target attributes. Extensive experiments on various kinds of visual attributes with SOTA personalization methods show the ability of the proposed method to mimic target visual appearance in novel contexts, thus improving the controllability and flexibility of personalization.
Paper Structure (27 sections, 4 equations, 13 figures, 2 tables)

This paper contains 27 sections, 4 equations, 13 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Personalization of diffution models
  4. Disentanglement in Generative Models
  5. Method
  6. Preliminaries for Initial Model
  7. Learning via Decoupled Self Augmentation
  8. LLM-Powered Prompt Generation.
  9. Initial Image Generation.
  10. Data Curation.
  11. Dual Concept Learning.
  12. Semantic Adjustment
  13. Experiments
  14. Experimental Setting
  15. Training Details.
  16. ...and 12 more sections

Figures (13)

  • Figure 1: Given a reference image, U-VAP can personalize the user-specified visual appearances and combine them with some novel concepts. U-VAP generates images with the material (left) or color (right) extracted from the input image of a rabbit statue (middle) and achieves better accuracy and controllability in several new concepts.
  • Figure 2: Pipeline of U-VAP. (a) Training: With input reference images, the initial concept-aware model is pre-trained on the entire concept. Meanwhile, given textual attribute query, U-VAP leverages GPT-3.5-turbo openaigpt35turbo to generate target and non-target descriptions for attribute modification. With these descriptions, U-VAP uses the initial concept-aware model to produce numerous candidate images and the data curation module filters them into target and non-target attribute sets. Subsequently, the identifiers $tgt$ and $ntg$ are optimized on each augmented set, which corresponds to the target and non-target attributes respectively. (b) Inference: we use semantic adjustment to correct the target embedding, further avoiding the entanglement of unwanted attributes in the generated results.
  • Figure 3: The workflow for decoupled self-augmentation. The "sks" represents the initial identifier learned in pre-learning step. With transformed query from guidance, an LLM openaigpt35turbo modifies the attribute descriptors and generates target and non-target descriptions respectively. Then the initial concept-aware model produces numerous candidate images. After data curation, U-VAP constructs attribute-aware samples for dual-concept learning.
  • Figure 4: Illustrations for semantic adjustment. By shifting semantic embedding on the adjustment direction, U-VAP can further promote the elimination of non-target attributes in the results.
  • Figure 5: Qualitative comparisons. Compared with SOTA personalization methods GPT-4V, ProSpect, DreamBooth and TI, we can achieve controlled and precise generation of specific visual attributes while maintaining high visual quality.
  • ...and 8 more figures