PathFormer: A Transformer with 3D Grid Constraints for Digital Twin Robot-Arm Trajectory Generation

TL;DR

PathFormer introduces a 3-grid trajectory representation (where/what/when) and constraint-masked decoding to enforce lattice-adjacent moves and workspace bounds, yielding robot-arm trajectories that are accurate, interpretable, and physically feasible. The method embeds geometry, task context, and temporal order into a Transformer, and uses a graph-constrained decoding process to guarantee validity of generated paths by construction, enabling robust sim-to-real transfer via a depth-camera digital twin. Empirically, PathFormer achieves high sequence fidelity (stepwise accuracy $\approx89.4\%$, F1 $\approx90.4\%$) and near-perfect validity ($\approx99.99\%$) on a 53k-trajectory corpus, and translates effectively to hardware with $\sim97.5\%$ reach and $\sim92.5\%$ pick success in central workspaces, plus $86.7\%$ end-to-end success in 60 cluttered-language tasks. This work demonstrates that embedding trajectory structure into Transformers bridges graph-based planning and long-horizon learning, offering a scalable, constraint-aware path to reliable manipulation and sim-to-real transfer.

Abstract

Robotic arms require precise, task-aware trajectory planning, yet sequence models that ignore motion structure often yield invalid or inefficient executions. We present a Path-based Transformer that encodes robot motion with a 3-grid (where/what/when) representation and constraint-masked decoding, enforcing lattice-adjacent moves and workspace bounds while reasoning over task graphs and action order. Trained on 53,755 trajectories (80% train / 20% validation), the model aligns closely with ground truth -- 89.44% stepwise accuracy, 93.32% precision, 89.44% recall, and 90.40% F1 -- with 99.99% of paths legal by construction. Compiled to motor primitives on an xArm Lite 6 with a depth-camera digital twin, it attains up to 97.5% reach and 92.5% pick success in controlled tests, and 86.7% end-to-end success across 60 language-specified tasks in cluttered scenes, absorbing slips and occlusions via local re-grounding without global re-planning. These results show that path-structured representations enable Transformers to generate accurate, reliable, and interpretable robot trajectories, bridging graph-based planning and sequence-based learning and providing a practical foundation for general-purpose manipulation and sim-to-real transfer.

Figures (7)

• Figure 1: Digital-twin sensing to grid. A RealSense depth camera provides $(x,y,z)$ targets on the local host; detections are projected onto a 3D grid used by PathFormer. The Lite 6 arm executes plans generated on this grid, enabling consistent sim-to-real transfer.
• Figure 2: End-to-end pipeline. Depth camera $\rightarrow$ 3D grid (origin, target, end-effector) $\rightarrow$PathFormer graph-constrained decoding $\rightarrow$ joint/gripper commands. The grid synchronizes the digital twin with the physical arm, while neighborhood masking enforces lattice adjacency.
• Figure 3: Perception-to-execution rollout.Left: RGB and depth streams from the RealSense D435i with YOLO-based object detection. Middle: Lite 6 robot executing reach--pick--place. Right: decoded 3D-grid trajectories for corresponding task phases. Together, these demonstrate how PathFormer integrates perception, digital twin alignment, and real-robot execution.
• Figure 4: Real-robot rollout and grid trajectories.Top: snapshots of reach--pick--place, showing end-effector motion across phases. Middle: normalized wrist IMU/gyro timeline, with annotations marking transitions between subtasks. Bottom: decoded 3D-grid paths (origin, target, end-effector) aligned with each phase. These results confirm that the digital twin mirrors physical execution while lattice adjacency constraints prevent discontinuities.
• Figure 5: Robot Arm hardware specifications and dimensions.