Prof. Robot: Differentiable Robot Rendering Without Static and Self-Collisions

TL;DR

This work addresses the gap in differentiable robot rendering by introducing a neural collision classifier coupled with an Eikonal regularizer to ensure gradient-consistent optimization, enabling collision-free action generation in SO(2)^K pose spaces. By integrating hierarchical pose encoding and SDF-like regularization, the approach avoids static and self-collisions without computing explicit manifold distances. The method demonstrates improved collision detection, reliable gradient-driven optimization, and seamless integration with Dr. Robot for image-driven control, trajectory generation, and VLM-based action distillation, with sim-to-real demonstrations validating practical applicability. Overall, Prof. Robot provides a scalable, memory-efficient plug-in for differentiable robotics that enhances physical realism and safety in learning-based manipulation.

Abstract

Differentiable rendering has gained significant attention in the field of robotics, with differentiable robot rendering emerging as an effective paradigm for learning robotic actions from image-space supervision. However, the lack of physical world perception in this approach may lead to potential collisions during action optimization. In this work, we introduce a novel improvement on previous efforts by incorporating physical awareness of collisions through the learning of a neural robotic collision classifier. This enables the optimization of actions that avoid collisions with static, non-interactable environments as well as the robot itself. To facilitate effective gradient optimization with the classifier, we identify the underlying issue and propose leveraging Eikonal regularization to ensure consistent gradients for optimization. Our solution can be seamlessly integrated into existing differentiable robot rendering frameworks, utilizing gradients for optimization and providing a foundation for future applications of differentiable rendering in robotics with improved reliability of interactions with the physical world. Both qualitative and quantitative experiments demonstrate the necessity and effectiveness of our method compared to previous solutions.

Figures (21)

• Figure 1: Our method improves upon Dr.Robot liu2024differentiable by additionally enabling differentiable avoidance of static and self-collisions. By learning and integrating a gradient-consistent pose classifier into a differentiable rendering pipeline, the generated pose trajectories are free from physical collisions. The objective is to penalize high collision probabilities during optimization. On the left, our method can push robotic arms away from intersecting objects. On the right, it can also avoid self-collisions. Images are rendered using Blender with meshes.
• Figure 2: Our Pipeline. We predict whether a collision has occurred based on pose input, using Eikonal loss ($\mathcal{L}_{\textrm{ek}}$) and binary cross-entropy loss ($\mathcal{L}_{\textrm{BCE}}$) to ensure this. It can then be seamlessly integrated with Dr. Robot liu2024differentiable to create a fully differentiable pipeline for optimization.
• Figure 3: An illustration of the potential fluctuation of the classification loss landscape.
• Figure 4: An illustration of how collision resolution optimizes a collided robot towards a collision-free state. The figure shown visualizes the SDF, where red points (representing collisions) are optimized to green points (representing non-collisions) along the gradient update trajectories.
• Figure 5: Sim2Real Experiment: Obstacle Avoidance with Prof. Robot. The first row illustrates that, in the simulator, the robot arm collides with the red box due to its inability to perceive obstacles. The second row shows that after integrating Prof. Robot, the robot successfully avoids the red obstacle. The red box represents an obstacle, and without Prof. Robot, the robot fails to circumvent it. The final row demonstrates that, with Prof. Robot, the robot smoothly navigates around the obstacle, confirming the effectiveness of our method for real-world applications.