ABPT: Amended Backpropagation through Time with Partially Differentiable Rewards

TL;DR

The paper addresses gradient bias when using first-order gradient methods with partially differentiable rewards in quadrotor control. It introduces ABPT, which amends BPTT by averaging a $0$-step return with an $N$-step return, augmented with value gradients, entropy regularization, and a state-replay initializer to encourage exploration. The approach yields faster convergence and higher final rewards across four quadrotor tasks in both simulation and real-world settings, outperforming PPO, BPTT, and SHAC baselines. This work advances differentiable physics-based learning by enabling robust, efficient policy optimization in the presence of non-differentiable reward components, with practical implications for real-world aerial robotics.

Abstract

Quadrotor control policies can be trained with high performance using the exact gradients of the rewards to directly optimize policy parameters via backpropagation-through-time (BPTT). However, designing a fully differentiable reward architecture is often challenging. Partially differentiable rewards will result in biased gradient propagation that degrades training performance. To overcome this limitation, we propose Amended Backpropagation-through-Time (ABPT), a novel approach that mitigates gradient bias while preserving the training efficiency of BPTT. ABPT combines 0-step and N-step returns, effectively reducing the bias by leveraging value gradients from the learned Q-value function. Additionally, it adopts entropy regularization and state initialization mechanisms to encourage exploration during training. We evaluate ABPT on four representative quadrotor flight tasks \li{in both real world and simulation}. Experimental results demonstrate that ABPT converges significantly faster and achieves higher ultimate rewards than existing learning algorithms, particularly in tasks involving partially differentiable rewards. The code will be released at http://github.com/Fanxing-LI/ABPT.

Figures (9)

• Figure 1: Our trained policies were validated in real world. Two rows are separately four trials of landing (Top) and hovering (Bottom). More videos are included in supplementary materials.
• Figure 2: An illustration for explaining biased gradient. In a racing tasks for drone which we introduce in Section \ref{['sec:taskDefine']}, the reward for passing the gate is a conditional constant, unable to automatically compute gradients.
• Figure 3: A overview of ABPT. ABPT combines 0-step return and n-step returns together, to compensate the biased gradient resulted by partially non-differentiable reward. The red dash lines indicate the direction of backpropagation. The replay buffer stores only visited states for episode initialization.
• Figure 4: Training curves of PPO, SHAC, BPTT, and our ABPT in both time-step (Top) and wall-time (Bottom). Each curve is averaged over results from five random seeds, and the shaded area denotes the range of best and worst reward.
• Figure 5: The curve shows the the difference between the parameters trained with fully differentiable and partially differentiable rewards. We deliberately detach parts of the rewards to interrupt gradient backpropagation and retrain the policy with or without combining the 0-step return.