Training-Free Text-to-Image Compositional Food Generation via Prompt Grafting

TL;DR

This work tackles object entanglement in compositional food image generation by introducing Prompt Grafting (PG), a training-free, two-stage diffusion sampling framework that first forms a separable layout using a layout prompt and then grafts to a content-focused target prompt once layout stability is reached. PG formalizes a time-dependent conditioning $ c(t)$, leverages $ \frac{dx}{dt} = f_\theta(x(t), t; c)$ dynamics, and uses CLIP-based grafting criteria to determine when to switch prompts, with optional spatial cues to boost separation. Empirically, PG improves object recall and existence on two food datasets (VFN and UEC-256) and demonstrates controllability over object co-location, while maintaining generalization to non-food domains; it remains training-free and annotation-free for layouts, making it practical for data augmentation and recipe visualization in dietary assessments. These findings highlight PG’s practical impact for reliable multi-item synthesis in real-world applications and suggest avenues for automatic spatial relation inference in future work.

Abstract

Real-world meal images often contain multiple food items, making reliable compositional food image generation important for applications such as image-based dietary assessment, where multi-food data augmentation is needed, and recipe visualization. However, modern text-to-image diffusion models struggle to generate accurate multi-food images due to object entanglement, where adjacent foods (e.g., rice and soup) fuse together because many foods do not have clear boundaries. To address this challenge, we introduce Prompt Grafting (PG), a training-free framework that combines explicit spatial cues in text with implicit layout guidance during sampling. PG runs a two-stage process where a layout prompt first establishes distinct regions and the target prompt is grafted once layout formation stabilizes. The framework enables food entanglement control: users can specify which food items should remain separated or be intentionally mixed by editing the arrangement of layouts. Across two food datasets, our method significantly improves the presence of target objects and provides qualitative evidence of controllable separation.

Figures (10)

• Figure 1: Example compositional food images generated by stable diffusion v3 model (SD3) and our method with corresponding reference images.
• Figure 2: Generated image from stable diffusion v1 and stable diffusion v3 model using text prompt: A photo of white rice and soup
• Figure 3: Generated image from stable diffusion v3 with input text prompt from "soup and grits" and "plate and bowl"
• Figure 4: Attention map for selected text tokens in the first 50 inference steps for Stable Diffusion v3 model given 100 total inference steps with text prompt input: "A photo of white rice and soup" Once layout formation stabilizes, later steps only refine details within the layout.
• Figure 5: Examples of image-caption data stable diffusion model pretrained on