Prompt2Craft: Generating Functional Craft Assemblies with LLMs

TL;DR

Prompt2Craft introduces the Craft Assembly Task, where an LLM-based decision agent infers an object subset and arrangements to assemble a functional craft that resembles a target image. A structured template constrains predictions, and the proposed pipeline validates proposals via format checks, collision tests, and physics-based simulations to ensure functional viability. The approach is evaluated across eight object categories and three target functions, comparing against a PartCrafter-based baseline using TripoSG-derived meshes as proxy ground truth for visual similarity. Results show solid functional success and competitive visual similarity, with ablation studies highlighting the importance of re-prompting strategies and the balance between collision/physics checks and design exploration.

Abstract

Inspired by traditional handmade crafts, where a person improvises assemblies based on the available objects, we formally introduce the Craft Assembly Task. It is a robotic assembly task that involves building an accurate representation of a given target object using the available objects, which do not directly correspond to its parts. In this work, we focus on selecting the subset of available objects for the final craft, when the given input is an RGB image of the target in the wild. We use a mask segmentation neural network to identify visible parts, followed by retrieving labeled template meshes. These meshes undergo pose optimization to determine the most suitable template. Then, we propose to simplify the parts of the transformed template mesh to primitive shapes like cuboids or cylinders. Finally, we design a search algorithm to find correspondences in the scene based on local and global proportions. We develop baselines for comparison that consider all possible combinations, and choose the highest scoring combination for common metrics used in foreground maps and mask accuracy. Our approach achieves comparable results to the baselines for two different scenes, and we show qualitative results for an implementation in a real-world scenario.

Figures (7)

• Figure 1: Concept of our method on using LLMs to generate craft assemblies which are validated in physics simulation for functionality.
• Figure 2: Overview of our approach using LLMs for the Craft Assembly Task. The inputs are used to populate a prompt for the LLM. The response of the LLM goes through multiple checks to generate a successful craft. If the proposal fails at any point, the LLM agent can be re-prompted.
• Figure 3: Qualitative results for the generated crafts. TripoSG directly infers the model from the masked image, while the PartCrafter* baseline and Ours (re-prompt w/o feedback) obtain a part separated model and chooses corresponding objects available in the scene. Ours generated more structured results.
• Figure 4: Failure cases of our method. Most of the failures are caused by collision between parts.
• Figure 5: JSON template format for one part of the assembly.