Parametric-ControlNet: Multimodal Control in Foundation Models for Precise Engineering Design Synthesis

TL;DR

This work presents Parametric-ControlNet, a multimodal conditioning framework that enables precise control of text-to-image foundation models for engineering design synthesis. It fuses parametric autocompletion, assembly-graph-based component encoding, CLIP text representations, and a ControlNet-style conditioner to steer diffusion outputs toward constraint-compliant designs. The system is validated on the BIKED bike-design dataset with extensive evaluations, including surrogate-model-based parametric accuracy, component conditioning, and multimodal generation metrics (PSNR, SSIM, IoC, and Diversity Score), showing substantial improvements over text-only baselines. The approach promises to enhance engineering design workflows by enabling robust multi-modal control, collaboration, and design exploration, with future work aimed at expanding modalities, domain applications, and rigorous evaluation benchmarks.

Abstract

This paper introduces a generative model designed for multimodal control over text-to-image foundation generative AI models such as Stable Diffusion, specifically tailored for engineering design synthesis. Our model proposes parametric, image, and text control modalities to enhance design precision and diversity. Firstly, it handles both partial and complete parametric inputs using a diffusion model that acts as a design autocomplete co-pilot, coupled with a parametric encoder to process the information. Secondly, the model utilizes assembly graphs to systematically assemble input component images, which are then processed through a component encoder to capture essential visual data. Thirdly, textual descriptions are integrated via CLIP encoding, ensuring a comprehensive interpretation of design intent. These diverse inputs are synthesized through a multimodal fusion technique, creating a joint embedding that acts as the input to a module inspired by ControlNet. This integration allows the model to apply robust multimodal control to foundation models, facilitating the generation of complex and precise engineering designs. This approach broadens the capabilities of AI-driven design tools and demonstrates significant advancements in precise control based on diverse data modalities for enhanced design generation.

Figures (8)

• Figure 1: Comparisons of features and generation results for different models. In Subfigure A, we show the additional control modalities and features that our model offers in comparison to the state-of-the-art models. in Subfigure B, we compare the performance in precise multimodal control and generation quality for different models.
• Figure 2: Overview of the ControlNet architecture.
• Figure 3: Overview of our model. We propose to add a parametric autocompletion and encoder conditioned on tabular data and assembly graphs, a component image encoder, a multimodal fusion module with attention layers to encode the parametric and textual information into a multimodal design parameter embedding.
• Figure 4: Comparisons of generated results from Stable Diffusion vs. our model when conditioned on component information. We show 4 samples for both models. The inputs for both models are text and component images. The results indicate that Stable Diffusion struggles to generate feasible bikes under this condition, while our model successfully generates valid bikes with greater diversity.
• Figure 5: Bike Renderings from Generated Parameters of the Imputation Model. Generated features are marked with a light blue tint. The input is testing parametric information with 10% of features masked. We then use the imputation model to generate the missing features. We show these hand-picked samples to show that the imputation model generates valid and diverse bike designs.