Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?

TL;DR

This work examines whether diffusion models can learn hidden inter-feature rules linking image features, revealing that DSM-trained models capture coarse relationships but struggle with fine-grained, precise rules. By constructing four synthetic tasks with controllable rule difficulty and providing a theoretical constant-error bound for DSM-based learning, the authors identify a fundamental limitation in rule conformity. They propose guided diffusion, via contrastive classifier guidance and pixel-space filtering, to mitigate these failures, achieving partial improvements but limited by weak signals and reliance on prior rule knowledge. The findings offer a principled understanding of the limitations of current diffusion training for rule-aligned generation and point to future directions for enabling finer-grained inter-feature reasoning.

Abstract

Despite the remarkable success of diffusion models (DMs) in data generation, they exhibit specific failure cases with unsatisfactory outputs. We focus on one such limitation: the ability of DMs to learn hidden rules between image features. Specifically, for image data with dependent features ($\mathbf{x}$) and ($\mathbf{y}$) (e.g., the height of the sun ($\mathbf{x}$) and the length of the shadow ($\mathbf{y}$)), we investigate whether DMs can accurately capture the inter-feature rule ($p(\mathbf{y}|\mathbf{x})$). Empirical evaluations on mainstream DMs (e.g., Stable Diffusion 3.5) reveal consistent failures, such as inconsistent lighting-shadow relationships and mismatched object-mirror reflections. Inspired by these findings, we design four synthetic tasks with strongly correlated features to assess DMs' rule-learning abilities. Extensive experiments show that while DMs can identify coarse-grained rules, they struggle with fine-grained ones. Our theoretical analysis demonstrates that DMs trained via denoising score matching (DSM) exhibit constant errors in learning hidden rules, as the DSM objective is not compatible with rule conformity. To mitigate this, we introduce a common technique - incorporating additional classifier guidance during sampling, which achieves (limited) improvements. Our analysis reveals that the subtle signals of fine-grained rules are challenging for the classifier to capture, providing insights for future exploration.

Figures (18)

• Figure 1: Synthetic Tasks Inspired by Real-World Insights. Based on whether inter-feature rules involve spatial dependencies, we categorize the failure cases into spatial and non-spatial rules. Spatial rules include: (a) Light-shadow, where evaluated DMs generate unreasonable multiple shadows or incorrect shadow flips; (b) Reflection/Refraction, showing incorrect mirror rules or missing refraction effects below water surface; (c) Semantics, such as inconsistencies between sunflower orientation and sun position, or brush and canvas colors. Non-spatial rules involve: (d) Size-Texture, like mismatches between tree diameter and growth rings; (e) Size/Region-Color, where evaluated models fail to capture burning candle's color variations and star size-color relationships (e.g., red giants and white dwarf); (f) Color-Color, as in Eclectus parrots' body-beak color correlations that DMs fail to maintain. \ref{['app:Details and More Example on Real-Wold Hidden Inter-Feature Rules']} provides detailed explanations for each case. These failures of mainstream DMs in handling real-world inter-feature rules inspire our design of four synthetic tasks.
• Figure 2: Pipeline for extracting features. Given an image, we first apply a color-based mask by using predefined colors, then count whether the number of masks meets expectations, and finally extract features of interest by marking the key points within masks.
• Figure 3: Synthetic training data satisfies fine-grained rules. To validate the evaluation method, we extract relevant features from the synthetic training data and check if they meet expectations, focusing on generations within the interval $[2.5\%,97.5\%]$ for stability. The closely matching Estimation and Ground Truth lines, along with an $R^2$ value near $1$, demonstrate effectiveness of the evaluation method.
• Figure 4: Generated data does not satisfy fine-grained rules. Considering generated samples within the $[2.5\%, 97.5\%]$ range, we extract focused features and check if they meet fine-grained rules. The Estimation line, far from the Ground Truth line, and an $R^2$ value less than $1$, reveal DMs' failure in learning fine-grained rules. \ref{['app:More Results of Synthetic Tasks']} shows generated images that violate the fine-grained rules.
• Figure 5: DMs generate rule-conforming samples. Define Rule-conforming generations have ratios (e.g., $\frac{l_2h_1}{l_1h_2}$ in Task A) within $\pm 0.01$ of true ratio ($1$ in Task A). \ref{['fig:rule_conforming']} shows DDPM's ability to generate rule-conforming samples across tasks. \ref{['fig:taska_memory_rates_13d']} indicates that nearest neighbor distances between $10$ rule-conforming samples in Task A and training data are large ($>0.3$), suggesting novel generation rather than memorization.

Theorems & Definitions (9)

• Definition 4.1: Data distribution with Inter-Feature Rules
• Theorem 4.2
• Definition 4.3: Rule-conforming error
• Theorem 4.4
• Theorem 4.5
• proof : Proof of Theorem \ref{['thm:score']}
• proof : Proof of Theorem \ref{['them:multi_poly']}
• Lemma 5.1: cao2022benign
• proof : Proof of Theorem \ref{['thm:main_linear']}