Beyond Binary Preference: Aligning Diffusion Models to Fine-grained Criteria by Decoupling Attributes

TL;DR

This work targets a fundamental bottleneck in diffusion-model alignment: coarse scalar or binary signals fail to capture fine-grained human expertise. It introduces a domain-specific, hierarchical evaluation for painting, encompassing 7 root dimensions across 246 attribute pairs, and annotates 10,277 paintings with A_pos and A_neg via a domain-expert agent. The authors then propose a two-stage framework: Supervised Fine-Tuning to create an expert model θ1 that encodes the fine-grained criteria, followed by Complex Preference Optimization (CPO) that uses dynamic winner/loser noise targets (z^w,z^l) from θ1 to steer a final model θ toward A_pos while avoiding A_neg, with a stabilization strategy L_CPO-S to balance gradients. Experiments on SDXL and FLUX show CPO reduces negative attributes, achieves superior FID and human-preference metrics, and scales to different architectures, confirming the practicality of fine-grained, domain-specific alignment for diffusion models in art generation. The approach promises stronger, more interpretable control over outputs by mirroring multi-dimensional expert evaluation, though it relies on surrogate expert models and domain-specific criteria, highlighting directions for broader generalization and bias mitigation.

Abstract

Post-training alignment of diffusion models relies on simplified signals, such as scalar rewards or binary preferences. This limits alignment with complex human expertise, which is hierarchical and fine-grained. To address this, we first construct a hierarchical, fine-grained evaluation criteria with domain experts, which decomposes image quality into multiple positive and negative attributes organized in a tree structure. Building on this, we propose a two-stage alignment framework. First, we inject domain knowledge to an auxiliary diffusion model via Supervised Fine-Tuning. Second, we introduce Complex Preference Optimization (CPO) that extends DPO to align the target diffusion to our non-binary, hierarchical criteria. Specifically, we reformulate the alignment problem to simultaneously maximize the probability of positive attributes while minimizing the probability of negative attributes with the auxiliary diffusion. We instantiate our approach in the domain of painting generation and conduct CPO training with an annotated dataset of painting with fine-grained attributes based on our criteria. Extensive experiments demonstrate that CPO significantly enhances generation quality and alignment with expertise, opening new avenues for fine-grained criteria alignment.

Figures (13)

• Figure 1: Existing methods rely on coarse-grained, scalar or binary image-level reward signals. In contrast, our method leverages human expert knowledge for fine-grained attribute decoupling, guiding the model directly from the noise space to approach positive and avoid negative directions.
• Figure 2: The pipeline of our framework. The Domain-Expert Agent decomposes image along 7 dimensions, which are represented as: Brushstroke and Texture, Light and Shadow, Shape and Posture, Composition, Perspective and Space, Color relationship, and Edge relationship. Notice that the visualization of the attribute hierarchy in the agent is simplified. The full hierarchy is of 5 levels with 246 attribute pairs in the leaf nodes. Post-annotation, we first conduct SFT to obtain the model $\theta_1$. This model is then used to dynamically acquire noise signals that aggregate decoupled attribute information. Subsequently, the aligned model is trained to learn the positive direction while suppressing the negative direction.
• Figure 3: Illustration of the CPO sampling trajectory. At each timestep $t$, CPO employs the expert model $\theta_1$ to provide deterministic positive and negative noise guidance, directing the trajectory toward virtual winning and losing samples, respectively. Owing to the determinism of the noise trajectory, the final sample $x_0$ can be precisely reconstructed back to $x_t$. Compared with original DPO, this design enables process-level guidance for model training rather than relying solely on the final endpoints, thereby making the training process more efficient.
• Figure 4: Visual comparison of different baselines and our CPO. #A_neg ($\downarrow$) and PickScore ($\uparrow$) are annotated in the lower-left and lower-right corners of each image, respectively. CPO outperforms all baselines in both negative-attribute avoidance and preference scoring.
• Figure 5: Curves of the win and lose parts of the loss function over training steps. The configuration with stabilization demonstrates significantly greater stability compared to the one without.