A3D: Adaptive Affordance Assembly with Dual-Arm Manipulation

TL;DR

A3D presents a dual-arm furniture assembly framework that learns adaptive, per-point affordances to identify stable support locations and dynamically adjust strategies via interaction feedback. It combines a dense geometric representation (PointNet++ features), a top-K affordance selection, a conditional variational autoencoder for action directions, and an interaction-context attention mechanism to refine decisions across long-horizon tasks. The method is trained with dedicated loss terms for affordance, proposal, and scoring, and is validated in an extended FurnitureBench/IsaacGym environment with 50+ parts across 8 furniture types, plus real-world robotics experiments that demonstrate robust generalization and adaptation. The results show superior performance over baselines and ablations, highlighting the importance of Top-K sampling and interaction-context adaptation for stable, cross-geometry dual-arm assembly with practical home-robot applications.

Abstract

Furniture assembly is a crucial yet challenging task for robots, requiring precise dual-arm coordination where one arm manipulates parts while the other provides collaborative support and stabilization. To accomplish this task more effectively, robots need to actively adapt support strategies throughout the long-horizon assembly process, while also generalizing across diverse part geometries. We propose A3D, a framework which learns adaptive affordances to identify optimal support and stabilization locations on furniture parts. The method employs dense point-level geometric representations to model part interaction patterns, enabling generalization across varied geometries. To handle evolving assembly states, we introduce an adaptive module that uses interaction feedback to dynamically adjust support strategies during assembly based on previous interactions. We establish a simulation environment featuring 50 diverse parts across 8 furniture types, designed for dual-arm collaboration evaluation. Experiments demonstrate that our framework generalizes effectively to diverse part geometries and furniture categories in both simulation and real-world settings.

Figures (8)

• Figure 1: Procedure of assembling a furniture (Row 1). Single Arm may not stably assemble parts and a second robot is then introduced. Before Interaction, part kinematics and dynamics, indicated by affordance, are ambiguous, and the interaction may fail. After Interaction, the adapted affordance proposes actions for stable support during assembly.
• Figure 2: Framework Overview. At each operation stage, the policy takes the point cloud and the selected action point as inputs to predict the support action. The robot moves the gripper to the recommended pose to support the assembly. If part displacement occurs—indicating insufficient support—the system logs the pre-support point cloud, executed action, and displacement as interaction context, then re-predicts the support action using the updated point cloud and accumulated context.
• Figure 3: Point-Level Adaptation Support Affordance Framework. The model completes support decisions by extracting visual features, computing Top-K point-level affordances and generating candidate directions, scoring and selecting point–direction pairs, and extracting interaction context features.
• Figure 4: Affordance Map. The figure displays affordance heatmaps generated for various objects in simulation before and after interaction. Red arrows indicate the direction of part movement, and circled regions denote the highest-scoring areas. In the subplots labeled “Concentrated,” it is evident that after interaction, high-scoring points converge more tightly at the correct locations; in the other subplots, the high-scoring points have shifted in accordance with the observed movement trends.
• Figure 5: Qualitative Analysis of Ablations. (Left) Without Top-K sampling, the robot fails to find robust manipulation points. (Right) Without interaction context, the robot lacks physical awareness to adjust its actions.