Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proportion and Perspective Control for Flow-Based Image Generation

Julien Boudier, Hugo Caselles-Dupré

TL;DR

This work introduces two specialized ControlNets for flow-based image generation (FLUX): Proportion ControlNet, which uses bounding boxes to guide object placement, and Perspective ControlNet, which uses vanishing lines to steer 3D geometry. Each module is trained independently with the base FLUX model frozen, initialized from a LineArt ControlNet, and learned via a conditional Flow Matching objective using vision-language annotated pipelines. The authors demonstrate that Proportion ControlNet can enforce spatial proportions and even interpret non-rectangular shapes, while Perspective ControlNet can handle 1- and 2-point perspectives but struggles with 3-point views due to data distribution and priors. They also show that joint control is feasible only with reduced guidance, highlighting trade-offs between fidelity and multi-constraint adherence, and provide open-source releases to encourage further development.

Abstract

While modern text-to-image diffusion models generate high-fidelity images, they offer limited control over the spatial and geometric structure of the output. To address this, we introduce and evaluate two ControlNets specialized for artistic control: (1) a proportion ControlNet that uses bounding boxes to dictate the position and scale of objects, and (2) a perspective ControlNet that employs vanishing lines to control the 3D geometry of the scene. We support the training of these modules with data pipelines that leverage vision-language models for annotation and specialized algorithms for conditioning image synthesis. Our experiments demonstrate that both modules provide effective control but exhibit limitations with complex constraints. Both models are released on HuggingFace: https://huggingface.co/obvious-research

Proportion and Perspective Control for Flow-Based Image Generation

TL;DR

This work introduces two specialized ControlNets for flow-based image generation (FLUX): Proportion ControlNet, which uses bounding boxes to guide object placement, and Perspective ControlNet, which uses vanishing lines to steer 3D geometry. Each module is trained independently with the base FLUX model frozen, initialized from a LineArt ControlNet, and learned via a conditional Flow Matching objective using vision-language annotated pipelines. The authors demonstrate that Proportion ControlNet can enforce spatial proportions and even interpret non-rectangular shapes, while Perspective ControlNet can handle 1- and 2-point perspectives but struggles with 3-point views due to data distribution and priors. They also show that joint control is feasible only with reduced guidance, highlighting trade-offs between fidelity and multi-constraint adherence, and provide open-source releases to encourage further development.

Abstract

While modern text-to-image diffusion models generate high-fidelity images, they offer limited control over the spatial and geometric structure of the output. To address this, we introduce and evaluate two ControlNets specialized for artistic control: (1) a proportion ControlNet that uses bounding boxes to dictate the position and scale of objects, and (2) a perspective ControlNet that employs vanishing lines to control the 3D geometry of the scene. We support the training of these modules with data pipelines that leverage vision-language models for annotation and specialized algorithms for conditioning image synthesis. Our experiments demonstrate that both modules provide effective control but exhibit limitations with complex constraints. Both models are released on HuggingFace: https://huggingface.co/obvious-research
Paper Structure (14 sections, 1 equation, 9 figures)

This paper contains 14 sections, 1 equation, 9 figures.

Table of Contents

  1. Introduction
  2. Method
  3. High-Level Conditioning Design
  4. Proportion Control via Bounding Boxes.
  5. Perspective Control via Vanishing Lines.
  6. Data Pipelines
  7. Proportion Pipeline.
  8. Perspective Pipeline.
  9. Experiments and Results
  10. Qualitative Results of Individual Modules
  11. Proportion Control.
  12. Perspective Control.
  13. Multi-ControlNet Composition
  14. Conclusion

Figures (9)

  • Figure 1: Overview of the proposed ControlNets for high-level artistic control.
  • Figure 2: Overview of the data pipelines. Both produce structured training triplets from unannotated datasets.
  • Figure 3: Results from the Proportion ControlNet.
  • Figure 4: Illustration of the stylistic bias. The horizontal axis represents the ControlNet guidance strength (0.5 to 0.9). The vertical axis represents the end percentage of the diffusion process during which the ControlNet is active (0.1 to 0.9). The "pictorial" effect intensifies as both values increase.
  • Figure 5: Results from the perspective ControlNet.
  • ...and 4 more figures