Digital-Twin Evaluation for Proactive Human-Robot Collision Avoidance via Prediction-Guided A-RRT*

TL;DR

This work tackles proactive safety in human–robot collaboration by forecasting granular, joint-level human motion and validating planned trajectories in a physics-based digital twin. It fuses RGB–D/IMU perception with a CNN–BiLSTM predictor, a capsule artificial potential field (APF) for risk scoring, and a GPU-accelerated A-RRT* replanner, all validated in a ROS 2 digital twin. Across 50 trials, it achieves 100% proactive collision avoidance with $>250\mathrm{mm}$ clearance and sub-2 s replanning, demonstrating substantial throughput and safety gains over kinodynamic planners. The digital twin enables safe, repeatable pre-deployment validation and latency bridging, though limitations include single-human assumptions and fixed capsule radii, motivating future hardware-in-the-loop and multi-user extensions.

Abstract

Human-robot collaboration requires precise prediction of human motion over extended horizons to enable proactive collision avoidance. Unlike existing planners that rely solely on kinodynamic models, we present a prediction-driven safe planning framework that leverages granular, joint-by-joint human motion forecasting validated in a physics-based digital twin. A capsule-based artificial potential field (APF) converts these granular predictions into collision risk metrics, triggering an Adaptive RRT* (A-RRT*) planner when thresholds are exceeded. The depth camera is used to extract 3D skeletal poses and a convolutional neural network-bidirectional long short-term memory (CNN-BiLSTM) model to predict individual joint trajectories ahead of time. A digital twin model integrates real-time human posture prediction placed in front of a simulated robot to evaluate motions and physical contacts. The proposed method enables validation of planned trajectories ahead of time and bridging potential latency gaps in updating planned trajectories in real-time. In 50 trials, our method achieved 100% proactive avoidance with > 250 mm clearance and sub-2 s replanning, demonstrating superior precision and reliability compared to existing kinematic-only planners through the integration of predictive human modeling with digital twin validation.

Figures (6)

• Figure 1: Digital-twin interface used for human-aware planning.
• Figure 2: End-to-end Digital-Twin planning pipeline for a Real to Simulaiton to Real.
• Figure 3: CNN–BiLSTM human-motion predictor. Input: 3 s (10 Hz) sequences of bone unit vectors and joint displacements for 15 joints. A 1D convolution stage captures short-term kinematics; stacked bidirectional LSTMs model long-horizon temporal dependencies. A fully connected decoder outputs 10-frame (1 s) displacement and bone-orientation predictions, reconstructed into full 3D poses.
• Figure 4: Test-split visualization after training. Blue points/links show the current measured 3D pose from a held-out test sequence; red shows the network’s 10-step (1 s) forecast in world coordinates.
• Figure 5: Baseline execution without predicted interference. (a) UR16e initiates the cycle. (b) Nominal shortest path while APF for all forecasted waypoints remains $<\tau$. (c) Task completes without replanning. The yellow curve shows the commanded trajectory.