Improving Physical Object State Representation in Text-to-Image Generative Systems

TL;DR

This work tackles the difficulty of accurately depicting object states (e.g., empty, absent, or negated conditions) in text-to-image generation. It introduces a fully automatic synthetic data pipeline that uses large language models and vision-language models to generate and filter prompts describing objects in varied states, producing 7600 high-quality image-text pairs. The authors fine-tune multiple open-source T2I models with LoRA adapters and demonstrate consistent gains in semantic alignment, as measured by GPT-based evaluation and VQA scores on GenAI-Object-State and Object State Bench, while maintaining visual quality. The results show substantial improvements in recognizing and rendering object states, with limited or no degradation on prompts unrelated to object states, and reveal the value of recaptioning and synthetic data as a data-centric approach to address semantic gaps in state representation.

Abstract

Current text-to-image generative models struggle to accurately represent object states (e.g., "a table without a bottle," "an empty tumbler"). In this work, we first design a fully-automatic pipeline to generate high-quality synthetic data that accurately captures objects in varied states. Next, we fine-tune several open-source text-to-image models on this synthetic data. We evaluate the performance of the fine-tuned models by quantifying the alignment of the generated images to their prompts using GPT4o-mini, and achieve an average absolute improvement of 8+% across four models on the public GenAI-Bench dataset. We also curate a collection of 200 prompts with a specific focus on common objects in various physical states. We demonstrate a significant improvement of an average of 24+% over the baseline on this dataset. We release all evaluation prompts and code.

Figures (16)

• Figure 1: Current text-to-image models struggle to depict common objects in varied physical states, inaccurately include unintended objects or fail to depict the requested empty or absence state (e.g., prompting for "A kitchen counter without any food" still results in a kitchen count full of food). Our method addresses these issues and yields accurate object state representation.
• Figure 2: State-of-the-art closed-source text-to-image and text-to-video models struggle to depict objects in absent or negation states. For Gen‑3 (a text-to-video model), we show a single extracted frame. This highlights the limitations of current advanced generative systems in accurately representing objects in simple and common physical states.
• Figure 3: Overview of the proposed synthetic data generation pipeline: We generate prompts describing common objects in different physical states. We next create images from the prompts, evaluate for the correct representation of the object state using GPT4o-mini hurst2024gpt. We rephrase prompts to introduce diversity in the sentence structures, length, and objects specified.
• Figure 4: System prompt used on the generated images for filtering out images not aligning with the provided prompt.
• Figure 5: Qualitative comparison of object state improvement for Stable Diffusion-1.5:(top) row shows the Stable Diffusion-1.5 baseline model, while the (bottom) row displays fine-tuned with our synthetic data pipeline, yielding more precise object state representation.