Differentiable Rendering as a Way to Program Cable-Driven Soft Robots

TL;DR

The paper addresses programming cable-driven soft robots by reframing tasks as differentiable rendering problems. A differentiable pipeline combines simulation, rendering, and objective computation to learn cable pull parameters via gradient-based optimization, using depth images from interior views to define gripping and avoidance losses. Key contributions include (i) a differentiable, physics-informed model of cable forces with barycentric mapping, (ii) a depth-image–driven objective framework for reaching, gripping, and obstacle avoidance, and (iii) demonstration across reach, avoidance, cylinder grasping, and eggshell gripping. This approach eliminates explicit landmark tracking and leverages gradient-based learning to configure soft robots in a digital twin, offering a lightweight path toward task specification and control via differentiable rendering. Future work points to integrating reinforcement learning for robustness across environments and extending the framework beyond predefined scenes.

Abstract

Soft robots have gained increased popularity in recent years due to their adaptability and compliance. In this paper, we use a digital twin model of cable-driven soft robots to learn control parameters in simulation. In doing so, we take advantage of differentiable rendering as a way to instruct robots to complete tasks such as point reach, gripping an object, and obstacle avoidance. This approach simplifies the mathematical description of such complicated tasks and removes the need for landmark points and their tracking. Our experiments demonstrate the applicability of our method.

Figures (6)

• Figure 1: The data flow of our differentiable pipeline. The simulation function $\mathcal{S}$ produces a state $\mathbf{s} _t$ using control parameters $\mathbf{p}$. The state is then projected to an image $\mathbf{I}_t$, which depending on the objective $\mathcal{O}$ is converted to a scalar loss $\ell$. The chain rule can then be applied to compute the derivative of $\ell$ w.r.t. control parameters $\mathbf{p}$.
• Figure 2: Left: A soft robot design whose fingers are curled by a cable running through them. Right: A depiction of how cable forces are computed for one finger. Cables in each segment are modeled with identical springs with damping to obtain forces at via points. Barycentric coordinates are then used to map these forces to nodal points.
• Figure 3: Didactic plots for depicting loss values given a robot depth map. Left: When gripping, the robot gets closer to the object over time and might penetrate it depending on the collision resolution. Right: The robot is assumed to be penetrating the obstacle initially and gets further away from it with time. Notice, how lower loss values would translate to gripping and avoiding in each case. In both cases, the object does not move, and as a result, its depth value remains constant.
• Figure 4: Snapshots of the scene showing the progression of tasks. First row: Reach Experiment at 0, 0.2, 2 seconds. Second row: Avoidance Experiment at 0, 0.2, 2 seconds. Third row: Cylinder Experiment at 0, 0.5, 3 seconds. Forth row: Egg Experiment at 0, 0.25, 2 seconds.
• Figure 5: Convergence plots for all experiments. First row: Reach Experiment. Second row: Avoidance Experiment. Third row: Cylinder Experiment. Fourth row: Egg Experiment. The columns show each cable's loss, parameter values, and gradients for each cable.