mrCAD: Multimodal Refinement of Computer-aided Designs

TL;DR

mrCAD introduces a large-scale, multimodal refinement dataset for computer-aided design, capturing two-player interactions where a Designer guides a Maker to iteratively reconstruct a target CAD. The dataset enables quantitative benchmarking via a programmatic distance metric and a gym-based evaluation framework, revealing distinctive language and drawing usage patterns between generation and refinement. Key findings show refinement markedly improves reconstruction and drawing contributes substantial geometric precision, while vision-language models underperform humans on refinement tasks. The work provides a foundation for modeling a multimodal refinement language and highlights the need for targeted training and data to close the gap in interactive CAD refinement tasks with AI systems.

Abstract

A key feature of human collaboration is the ability to iteratively refine the concepts we have communicated. In contrast, while generative AI excels at the \textit{generation} of content, it often struggles to make specific language-guided \textit{modifications} of its prior outputs. To bridge the gap between how humans and machines perform edits, we present mrCAD, a dataset of multimodal instructions in a communication game. In each game, players created computer aided designs (CADs) and refined them over several rounds to match specific target designs. Only one player, the Designer, could see the target, and they must instruct the other player, the Maker, using text, drawing, or a combination of modalities. mrCAD consists of 6,082 communication games, 15,163 instruction-execution rounds, played between 1,092 pairs of human players. We analyze the dataset and find that generation and refinement instructions differ in their composition of drawing and text. Using the mrCAD task as a benchmark, we find that state-of-the-art VLMs are better at following generation instructions than refinement instructions. These results lay a foundation for analyzing and modeling a multimodal language of refinement that is not represented in previous datasets.

Figures (9)

• Figure 1: We present a dataset of multimodal instructions for creating and modifying computer aided designs, along accompanying executions of these instructions in a 2D CAD environment. Participants worked together to recreate a target CAD over multiple rounds. Crucially, the target design is known only to the designer, who must instruct the maker on what to create. The maker, in turn, must edit the CAD model based on these instructions.
• Figure 2: The asymmetric chamfer distance from CAD $D$ to CAD $E$ is calculated by sampling 10 points on every curve of $D$, and calculating (symbolically) the minimum distance from each point to $E$. Each distance is then normalized by multiplying $\frac{1}{4}$ of the maximum size of the canvas, making it invariant to the canvas size. These distances are summed, making the asymmetrical chamfer distance. The symmetric Chamfer distance we use is the average of both directions.
• Figure 3: A: reconstruction accuracy for the 4 communication conditions --- multimodal+refinement, text only + refinement, drawing only + refinement, and multimodal + generation only. Using text only was less effective. Having no refinement was less effective. With refinement, drawing only and multimodal are comparable in performance. B: usage of text across rounds --- in the multimodal condition, participants used more texts in the later refinement rounds, suggesting a usage of text in conjunction with drawings to communicate refinements. C: usage of drawing across rounds --- in the multi-modal condition, participants used more drawing in the generation round, and less in the refinement rounds.
• Figure 4: A The mrCAD dataset contains three subsets: the coverage set of 2249 CADs with 1-2 successful rollouts, dense set of 698 CADs with 3+ successful reconstruction, and the very-dense set of 27 CADs with 30+ successful reconstruction. B We implemented a dynamic threshold for submitting designs that became more lenient in later rounds. Participants took a variable number of rounds to reach the threshold. Visualizing distance to the target broken down by round submitted reveals a trend of refinement over time. Red dashed line indicates the fixed threshold for including in analysis.
• Figure 5: A Example rollouts from the dataset. Target CADs (top-center) were shown to Designers, who created instructions (left columns) that Makers followed (right columns). Dyads iteratively refined their CADs across a series of rounds (rows). B Examples of multimodal refinement instructions. Language and drawing mutually constrain and inform the others' semantics. Many instructions don't make sense without the accompanying drawings, and vice-versa.