Sketch2Prototype: Rapid Conceptual Design Exploration and Prototyping with Generative AI

TL;DR

The paper addresses rapid conceptual design exploration by transforming hand-drawn sketches into text descriptions, diverse 2D images, and 3D prototypes ready for fabrication. It introduces a multi-stage pipeline—sketch-to-text via GPT-4V, text-to-image via DALL-E 3, and image-to-3D via multiple predictors—with Blender-based post-processing and additive manufacturing. Results show that employing text as an intermediate modality yields more diverse and manufacturable 3D designs than direct sketch-to-3D baselines, while also enabling user-driven iterative refinement through prompts. An open-source dataset of 1,087 milk-frother sketches with 4 generated images per sketch (4,348 images total) demonstrates alignment and diversity, underscoring the practical potential of AI-assisted design workflows and the value of human-in-the-loop feedback in early-stage prototyping.

Abstract

Sketch2Prototype is an AI-based framework that transforms a hand-drawn sketch into a diverse set of 2D images and 3D prototypes through sketch-to-text, text-to-image, and image-to-3D stages. This framework, shown across various sketches, rapidly generates text, image, and 3D modalities for enhanced early-stage design exploration. We show that using text as an intermediate modality outperforms direct sketch-to-3D baselines for generating diverse and manufacturable 3D models. We find limitations in current image-to-3D techniques, while noting the value of the text modality for user-feedback and iterative design augmentation.

Figures (7)

• Figure 1: The Sketch2Prototype framework takes in a conceptual design sketch as input, and produces multiple inspired images, a 3D model, and finally a fabricated prototype.
• Figure 2: The Sketch2Prototype framework uses transformer-based models for the sketch-to- text and text-to-image steps, as well as an encoder and conditional diffusion model for image- to-3D model. Post-processing of the 3D model is performed in Blender.
• Figure 3: Our framework enables exploration of the design space by automatically generating multiple diverse images inspired by one sketch. Here a single sketch results in three fabricated looks-like prototypes.
• Figure 4: Examples of the full Sketch2Prototype framework for four different design types: rodeo-inspired milk frother, pen-and-coin holder, phone stand, and backpack-inspired mug.
• Figure 5: A: 3D models generated from varying input images: Sketch2Prototype generates more diverse and manufacturable designs. B: The text modality allows for user control. We append text to the original prompt to generate different designs.