Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation

TL;DR

This work introduces statistical metrics that quantify both the linguistic and visual skew of a dataset for relational learning, and shows that generalization failures of text-to-image generation are a direct result of incomplete or unbalanced phenomenological coverage.

Abstract

The literature on text-to-image generation is plagued by issues of faithfully composing entities with relations. But there lacks a formal understanding of how entity-relation compositions can be effectively learned. Moreover, the underlying phenomenon space that meaningfully reflects the problem structure is not well-defined, leading to an arms race for larger quantities of data in the hope that generalization emerges out of large-scale pretraining. We hypothesize that the underlying phenomenological coverage has not been proportionally scaled up, leading to a skew of the presented phenomenon which harms generalization. We introduce statistical metrics that quantify both the linguistic and visual skew of a dataset for relational learning, and show that generalization failures of text-to-image generation are a direct result of incomplete or unbalanced phenomenological coverage. We first perform experiments in a synthetic domain and demonstrate that systematically controlled metrics are strongly predictive of generalization performance. Then we move to natural images and show that simple distribution perturbations in light of our theories boost generalization without enlarging the absolute data size. This work informs an important direction towards quality-enhancing the data diversity or balance orthogonal to scaling up the absolute size. Our discussions point out important open questions on 1) Evaluation of generated entity-relation compositions, and 2) Better models for reasoning with abstract relations.

Figures (17)

• Figure 1: Example images generated by DALL·E3. In all three cases, entities and relations are common but their compositions are uncommon. DALL·E3 tends to (a) compose entities unnaturally, (b) get trapped by the canonical relation, or (c) disregard the requested ordering. These errors are recurring across multiple trials, suggesting that DALL·E3 does not grasp the abstract notion of relations.
• Figure 2: Conceptual Framework. Text-to-Image generation consists of three important distinct components: A text encoder, a visual decoder, and a mechanism to communicate between these two spaces. Generation of images with consistent spatial relations requires that 1) the text encoder distinctly encodes linguistic roles, 2) the image generator distinguishes spatial roles in the output space, and 3) learning the correct translation from linguistic roles to visual roles. Suppose pre-training or architectural expressivity can fulfill the first two requirements, the remaining core task is to learn an effective communication channel -- often instantiated as cross-attention in diffusion models. To this end, we propose statistical metrics to formally quantify how the training data distribution received by the communication channel affects generalization.
• Figure 3: Sketched illustrations of phenomenological coverage with different properties. Shaded areas represent the training set, while blank areas represent the testing set. Columns and rows are organized by the concepts bound to position 1 and position 2 respectively. For example, the black cell in (a) represents the training instance ($c_2/p_1, c_3/p_2$), the red cell in (a) represents the testing instance ($c_9/p_1, c_1/p_2$). (a) Both positions are incomplete (b) Only $p_1$ is incomplete (c) Complete but unbalanced (d) Complete but unbalanced (e) Complete but unbalanced (f) Complete and balanced (g) Complete and balanced (h) Complete and balanced
• Figure 4: Training, testing and evaluation pipeline. We train diffusion models to generate images of two concepts ($c_1$, $c_2$) with a specified spatial relation. Then the model is tested on unseen concept pairs to see whether the learned relations are generalizable.
• Figure 5: Three types of learning dynamics are observed in our experiments. In the worst scenario (left), the testing accuracy plateaus and never converges. In the best scenario (right), the testing accuracy closely tracks the training accuracy until both converge to perfect. In the middle scenario (center), the testing accuracy climbs slower than the training accuracy, but is still able to converge to perfect at a delayed point after the training accuracy has already converged. In order to distinguish between the middle and best scenarios, we additionally report the accumulative gap between training and testing accuracy curves, which captures the timeliness of generalization.