CRAFT: Designing Creative and Functional 3D Objects

TL;DR

ShapeCraft addresses the problem of designing 3D objects that are simultaneously body-aware and semantically-guided. It introduces a Jacobian-based deformation framework that assigns per-triangle Jacobians $J_i \in \mathbb{R}^{3\times3}$ and computes a deformation map $\Phi^*$ by solving a Poisson-like energy, enabling joint optimization for text/image guidance and body contact. The method yields semantically aligned, body-fitting shapes across diverse object categories and body shapes, outperforming two-stage baselines in alignment and penetration metrics and enabling fabrication and sketch-guided workflows. By integrating differentiable rendering, CLIP-based guidance, and deformation Jacobians, ShapeCraft streamlines the creation of personalized, wearable 3D assets for both virtual avatars and real-world use, with broad implications for rapid ideation and customization in design. Overall, ShapeCraft demonstrates a practical pathway to combine semantics and body-awareness in 3D object design without requiring manual intervention.

Abstract

For designing a wide range of everyday objects, the design process should be aware of both the human body and the underlying semantics of the design specification. However, these two objectives present significant challenges to the current AI-based designing tools. In this work, we present a method to synthesize body-aware 3D objects from a base mesh given an input body geometry and either text or image as guidance. The generated objects can be simulated on virtual characters, or fabricated for real-world use. We propose to use a mesh deformation procedure that optimizes for both semantic alignment as well as contact and penetration losses. Using our method, users can generate both virtual or real-world objects from text, image, or sketch, without the need for manual artist intervention. We present both qualitative and quantitative results on various object categories, demonstrating the effectiveness of our approach.

Figures (14)

• Figure 1: ShapeCraft generates 3D shapes given text as input. The objects are optimized to fit on various character body shapes.
• Figure 2: Our method generates a variety of semantics and body-aware objects from input text prompts.
• Figure 3: Our method can deform the same template mesh into different text-specified geometries that are body-fitting.
• Figure 4: Method overview. We synthesize body-aware 3D objects from a base mesh given an input body geometry and either text or image as guidance. We propose to use a mesh deformation procedure that optimizes for both semantic alignment as well as contact and penetration losses.
• Figure 5: We show the effect of contact vs. penetration losses on text guided deformation for "cat mask".