Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Models

TL;DR

Manual2Skill++ tackles the last-mile challenge in robotic assembly by treating connectors as first-class primitives and extracting structured, connector-aware representations from instruction manuals using vision-language models. It introduces a universal hierarchical graph representation with explicit connector edges and a two-stage VLM-based graph generation pipeline that outputs precise attachment-point pairings, enabling direct pose optimization. A constraint-based pose alignment module delivers millimeter-level accuracy, bridging high-level task understanding with low-level manipulation, while a four-task Isaac Lab benchmark and a 21-task hierarchical-graph dataset validate generalization across furniture, toys, and industrial components under realistic connection modalities. The work advances connector-aware manipulation, multimodal task understanding, and the development of generalizable assembly systems with practical impact for real-world manufacturing and assembly tasks.

Abstract

Assembly hinges on reliably forming connections between parts; yet most robotic approaches plan assembly sequences and part poses while treating connectors as an afterthought. Connections represent the critical "last mile" of assembly execution, while task planning may sequence operations and motion plan may position parts, the precise establishment of physical connections ultimately determines assembly success or failure. In this paper, we consider connections as first-class primitives in assembly representation, including connector types, specifications, quantities, and placement locations. Drawing inspiration from how humans learn assembly tasks through step-by-step instruction manuals, we present Manual2Skill++, a vision-language framework that automatically extracts structured connection information from assembly manuals. We encode assembly tasks as hierarchical graphs where nodes represent parts and sub-assemblies, and edges explicitly model connection relationships between components. A large-scale vision-language model parses symbolic diagrams and annotations in manuals to instantiate these graphs, leveraging the rich connection knowledge embedded in human-designed instructions. We curate a dataset containing over 20 assembly tasks with diverse connector types to validate our representation extraction approach, and evaluate the complete task understanding-to-execution pipeline across four complex assembly scenarios in simulation, spanning furniture, toys, and manufacturing components with real-world correspondence.

Figures (4)

• Figure 1: Manual2Skill++. We extract connection information from assembly manuals, build connection-enriched hierarchical graphs encoding part relationships and connector details, then execute precise robotic assembly guided by these connection relations.
• Figure 2: Framework Overview. (1) A VLM processes manual images to extract part and connector relations, generating a connection-enriched hierarchical assembly representation. (2) The extracted connection constraints guide a geometric optimization process to compute precise target poses for parts and connectors. (3) The system executes the assembly by planning and performing robotic actions based on the connection-enriched hierarchical graph and aligned poses.
• Figure 3: Overview of Our Dataset and Benchmark. The dataset comprises 21 representative assembly tasks, each with high-fidelity 3D part models, manual pages for every step, and fully annotated connection information. From this collection, we selected four tasks and implemented them in Isaac Lab, designing connection mechanics to enable long-horizon assembly with explicit connector operations that directly correspond to real-world procedures.
• Figure 4: Real-World Correspondence of Our Benchmark Tasks. Top: SUNDVIK chair and VASSKAR shelf assembled from IKEA retail kits. Bottom: LEGO figure and airplane model assembled from 3D-printed parts matching simulation geometry. The one-to-one correspondence in part geometry, connector types, and assembly sequences validates our benchmark as a faithful testbed for real-world assembly research.