Component Selection for Craft Assembly Tasks

TL;DR

The paper introduces the Craft Assembly Task, a flexible robotic assembly problem in which a target object is represented using non-exact parts drawn from a scene based on a single RGB image. It proposes a four-stage pipeline that leverages template mesh retrieval, differentiable pose optimization, symmetry-based occlusion completion, and a proportion-driven scene-matching search to map parts to scene objects as cuboids or cylinders. The authors demonstrate that their template-mesh–driven approach achieves performance comparable to baselines across multiple metrics and validate it in a real-world robotic setup, underscoring practical feasibility. This work reduces data requirements for 3D part assemblies and offers a framework for open-ended object assembly with non-exact part correspondences, with potential extensions to 3D reasoning via LLMs.

Abstract

Inspired by traditional handmade crafts, where a person improvises assemblies based on the available objects, we formally introduce the Craft Assembly Task. It is a robotic assembly task that involves building an accurate representation of a given target object using the available objects, which do not directly correspond to its parts. In this work, we focus on selecting the subset of available objects for the final craft, when the given input is an RGB image of the target in the wild. We use a mask segmentation neural network to identify visible parts, followed by retrieving labelled template meshes. These meshes undergo pose optimization to determine the most suitable template. Then, we propose to simplify the parts of the transformed template mesh to primitive shapes like cuboids or cylinders. Finally, we design a search algorithm to find correspondences in the scene based on local and global proportions. We develop baselines for comparison that consider all possible combinations, and choose the highest scoring combination for common metrics used in foreground maps and mask accuracy. Our approach achieves comparable results to the baselines for two different scenes, and we show qualitative results for an implementation in a real-world scenario.

Figures (5)

• Figure 1: Illustrative example of a traditional craft task. Even if the available objects are non-exact correspondences to the target, a human is capable of abstracting and manipulating these objects to obtain a craft similar to their target.
• Figure 2: Overview of our proposed solution for the Craft Assembly Task. Given the RGB image of the target object, the visible parts are segmented and classified. Labeled template meshes of the detected class are retrieved from a prepared database and their pose are optimized through a differentiable renderer using the segmentation results as the target. The parts of the best aligned mesh are simplified to primitive shapes. Finally each part is matched with an object in the scene (input) using a search algorithm, generating the final Craft Proposal.
• Figure 3: Renders of the final combinations for the baselines and our method. We also show the evaluation results for each metric for our proposed craft.
• Figure 4: Qualitative examples of our method applied to novel instances. Although the final craft cannot match the input, due to structural differences in our template meshes, the segmentation and pose can still be retrieved.
• Figure 5: Example of implementation in a real world environment for the input image of a bus. During assembly, some adjustments to avoid collision are done through pre-defined rules. The render area is maximized for better visualization. A video demonstration for this input is available at: https://youtu.be/tjz2d_NuxB8