MCD: A Model-Agnostic Counterfactual Search Method For Multi-modal Design Modifications

TL;DR

MCD reframes design modification as a model-agnostic, multi-objective counterfactual search, decoupling optimization from sampling to efficiently generate minimal, realistic design changes. It extends prior counterfactual work by supporting hard and soft constraints, manifold proximity, and multi-modal requirements (text/image prompts) via CLIP embeddings. The approach is demonstrated through bicycle-design case studies that show improvements in structural efficiency, cross-modal aesthetic alignment, and rider-specific ergonomics, while maintaining feasible modification magnitudes. The work provides a practical, scalable tool for designers to explore targeted, multi-faceted design modifications and offers publicly available code, data, and demos to facilitate adoption. This contributes a versatile framework for rapid, human-centric design iteration across engineering domains.

Abstract

Designers may often ask themselves how to adjust their design concepts to achieve demanding functional goals. To answer such questions, designers must often consider counterfactuals, weighing design alternatives and their projected performance. This paper introduces Multi-objective Counterfactuals for Design (MCD), a computational tool that automates and streamlines the counterfactual search process and recommends targeted design modifications that meet designers' unique requirements. MCD improves upon existing counterfactual search methods by supporting multi-objective requirements, which are crucial in design problems, and by decoupling the counterfactual search and sampling processes, thus enhancing efficiency and facilitating objective trade-off visualization. The paper showcases MCD's capabilities in complex engineering tasks using three demonstrative bicycle design challenges. In the first, MCD effectively identifies design modifications that quantifiably enhance functional performance, strengthening the bike frame and saving weight. In the second, MCD modifies parametric bike models in a cross-modal fashion to resemble subjective text prompts or reference images. In a final multidisciplinary case study, MCD tackles all the quantitative and subjective design requirements introduced in the first two problems, while simultaneously customizing a bike design to an individual rider's biomechanical attributes. By exploring hypothetical design alterations and their impact on multiple design objectives, MCD recommends effective design modifications for practitioners seeking to make targeted enhancements to their designs. The code, test problems, and datasets used in the paper are available to the public at decode.mit.edu/projects/counterfactuals/.

Figures (6)

• Figure 1: Multi-objective Counterfactuals for Design answers "Inverse Counterfactual" design questions --- Given an existing design and its properties, as well as a set of target properties (requirements) and property predictors (design evaluators), MCD identifies variants of the design that achieve the requirements. This enables a human-AI collaborative workflow in which MCD recommends efficient design modifications to achieve designer-specified goals.
• Figure 2: Demonstration of inverse counterfactual search in 2D space. The relative priority weighting of objectives has a significant impact on counterfactuals sampled during sampling.
• Figure 3: Visualization of the objective manifold for cross-modal counterfactual selection. Designs sampled from the top of the manifold prioritize proximity, sparsity, and manifold proximity. Designs on the left and right sides prioritize similarity to a text prompt and reference image, respectively. Note: Designs are optimized parametrically by modifying CAD features and are rendered for visualization purposes. Objectives are calculated in a cross-modal fashion.
• Figure 4: Pairplots visualizing objective score distributions for identified counterfactuals and the dataset. Individual kernel density estimates are shown on the diagonal, while pairwise scatterplots are shown on the off-diagonal. The counterfactual query's objective scores are marked with a black X. We also show sparsity, proximity, and manifold proximity score distributions over the counterfactuals.
• Figure 5: Comparison of bikes generated by MCD and bikes generated by classic optimization. Optimization is heavily reliant on explicit and exhaustive constraints and tends to fail in their absence. In contrast, counterfactual search can implicitly obey unknown constraints through its proximity, sparsity, and manifold proximity objectives.