From Perception to Symbolic Task Planning: Vision-Language Guided Human-Robot Collaborative Structured Assembly

TL;DR

This work tackles robust state estimation and adaptive task planning for human-robot collaboration in structured assembly by grounding perception in design priors and coupling it with symbolic, knowledge-driven planning. The authors propose two modules: Perception-to-Symbolic State (PSS), which converts RGB-D observations into a verified symbolic state $U_t$ using a design image $I^{des}$ and a multidimensional component array $\mathbf{A}$; and Human-Aware Planning and Replanning (HPR), which performs frontier-based task allocation and updates plans $P_t$ with a minimal-change rule to preserve stability under human interventions. Key contributions include (i) a design-grounded symbolic state synthesis method that leverages VLMs with deterministic rule checks, (ii) a knowledge-driven, human-aware planning framework that maintains plan validity and minimizes disruption, and (iii) a timber-frame case study showing 97% PSS state-synthesis accuracy and robust planning across dynamic HRC scenarios. The framework enhances robustness of state estimation and task planning in real-world, dynamic assembly sites and can be extended with multi-view sensing and learning-based rule induction to further improve scalability and generalization.

Abstract

Human-robot collaboration (HRC) in structured assembly requires reliable state estimation and adaptive task planning under noisy perception and human interventions. To address these challenges, we introduce a design-grounded human-aware planning framework for human-robot collaborative structured assembly. The framework comprises two coupled modules. Module I, Perception-to-Symbolic State (PSS), employs vision-language models (VLMs) based agents to align RGB-D observations with design specifications and domain knowledge, synthesizing verifiable symbolic assembly states. It outputs validated installed and uninstalled component sets for online state tracking. Module II, Human-Aware Planning and Replanning (HPR), performs task-level multi-robot assignment and updates the plan only when the observed state deviates from the expected execution outcome. It applies a minimal-change replanning rule to selectively revise task assignments and preserve plan stability even under human interventions. We validate the framework on a 27-component timber-frame assembly. The PSS module achieves 97% state synthesis accuracy, and the HPR module maintains feasible task progression across diverse HRC scenarios. Results indicate that integrating VLM-based perception with knowledge-driven planning improves robustness of state estimation and task planning under dynamic conditions.

Figures (13)

• Figure 1: Overview of the proposed planning framework, showing the interaction between the Perception-to-Symbolic State module and Human-Aware Planning and Replanning module in a closed-loop perception to task planning process.
• Figure 2: Overview of the Perception-to-Symbolic State module, which synthesizes a verified symbolic assembly state from design specifications and RGB-D observations..
• Figure 3: Structure of the multidimensional component array $\mathbf{A}$.
• Figure 4: Design-aligned perceptual representation pipeline for converting raw RGB-D observations into structured geometric primitives.
• Figure 5: Example layout used for the case study. The configuration includes two heterogeneous collaborative robots, an overhead RGB-D camera, and the design coordinate frame registered to the workspace.