FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects

TL;DR

FlowBot3D introduces 3D Articulation Flow (3DAF), a per-point motion representation for articulated objects, and the ArtFlowNet predictor that maps depth-based point clouds to 3DAF. The system uses a general, two-phase policy to grasp at the strongest predicted motion and to iteratively follow flow directions, enabling articulation across unseen object categories. Experimental results in simulation (PartNet-Mobility via ManiSkill) and real-world Sawyer robot experiments demonstrate state-of-the-art performance and strong sim-to-real transfer, with notable robustness to occlusions and object variability. The work highlights a principled separation between affordance learning and motion planning, achieving broad generalization without explicit articulation-parameter supervision.

Abstract

We explore a novel method to perceive and manipulate 3D articulated objects that generalizes to enable a robot to articulate unseen classes of objects. We propose a vision-based system that learns to predict the potential motions of the parts of a variety of articulated objects to guide downstream motion planning of the system to articulate the objects. To predict the object motions, we train a neural network to output a dense vector field representing the point-wise motion direction of the points in the point cloud under articulation. We then deploy an analytical motion planner based on this vector field to achieve a policy that yields maximum articulation. We train the vision system entirely in simulation, and we demonstrate the capability of our system to generalize to unseen object instances and novel categories in both simulation and the real world, deploying our policy on a Sawyer robot with no finetuning. Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments.

Figures (8)

• Figure 1: FlowBot3D in action. The system first observes the initial configuration of the object of interest, estimates the per-point articulation flow of the point cloud (3DAF), then executes the action based on the selected flow vector. Here, the red vectors represent the direction of flow of each point (object points appear in blue); the magnitude of the vector corresponds to the relative magnitude of the motion that point experiences as the object articulates.
• Figure 2: Illustrations of prismatic and revolute joints.
• Figure 3: FlowBot3D System Overview. Our system in deployment has two phases: the Grasp-Selection phase and the Articulation-Execution Phase. The dark red dots represent the predicted location of each point, and the light red lines represent the flow vectors connecting from the current time step's points to the predicted points. Note that the flow vectors are downsampled for visual clarity. In Grasp-Selection Phase, the agent observes the environment in the format of point cloud data. The point cloud data will then be post-processed and fed into the ArtFlowNet, which predicts per-point 3D flow vectors. The system then chooses the point that has the maximum flow vector magnitude and deploys motion planning to make contact with the chosen point using suction. In Articulation-Execution phase, after making suction contact with the chosen argmax point, the system iteratively observes the pointcloud data and predicts the 3D flow vectors. In this phase, the motion planning module would guide the robot to follow the maximum observable flow vector's direction and articulate the object of interest repeatedly.
• Figure 4: Workspace setup for physical experiments. The sensory signal comes from an Azure Kinect depth camera, and the agent is a Sawyer BLACK robot.
• Figure 5: Fourteen test objects for our real-world experiments. Please refer to Supplementary Material for the exact category of each object.