Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning passive policies with virtual energy tanks in robotics

Riccardo Zanella, Gianluca Palli, Stefano Stramigioli, Federico Califano

TL;DR

The paper addresses stability and safety challenges in learning-based robotics by fusing passivity-based control with reinforcement learning through virtual energy tanks. It introduces discrete-time energy sampling and two passivization modes—inference and training—to ensure passive closed-loop behavior while preserving learning capabilities, supported by both DoorOpening and Pendulum simulations. Key contributions include a formal energy-tank framework, task-energy concepts such as $e^*$ and $e_0$, and empirical demonstrations that passive-wrapped policies retain performance under disturbances while enabling energy-aware training. This work advances energy-aware robotics by providing practical methods to combine formal stability with data-driven control and suggests future directions for safety-aware, energy-constrained learning across more complex tasks.

Abstract

Within a robotic context, we merge the techniques of passivity-based control (PBC) and reinforcement learning (RL) with the goal of eliminating some of their reciprocal weaknesses, as well as inducing novel promising features in the resulting framework. We frame our contribution in a scenario where PBC is implemented by means of virtual energy tanks, a control technique developed to achieve closed-loop passivity for any arbitrary control input. Albeit the latter result is heavily used, we discuss why its practical application at its current stage remains rather limited, which makes contact with the highly debated claim that passivity-based techniques are associated with a loss of performance. The use of RL allows us to learn a control policy that can be passivized using the energy tank architecture, combining the versatility of learning approaches and the system theoretic properties which can be inferred due to the energy tanks. Simulations show the validity of the approach, as well as novel interesting research directions in energy-aware robotics.

Learning passive policies with virtual energy tanks in robotics

TL;DR

The paper addresses stability and safety challenges in learning-based robotics by fusing passivity-based control with reinforcement learning through virtual energy tanks. It introduces discrete-time energy sampling and two passivization modes—inference and training—to ensure passive closed-loop behavior while preserving learning capabilities, supported by both DoorOpening and Pendulum simulations. Key contributions include a formal energy-tank framework, task-energy concepts such as and , and empirical demonstrations that passive-wrapped policies retain performance under disturbances while enabling energy-aware training. This work advances energy-aware robotics by providing practical methods to combine formal stability with data-driven control and suggests future directions for safety-aware, energy-constrained learning across more complex tasks.

Abstract

Within a robotic context, we merge the techniques of passivity-based control (PBC) and reinforcement learning (RL) with the goal of eliminating some of their reciprocal weaknesses, as well as inducing novel promising features in the resulting framework. We frame our contribution in a scenario where PBC is implemented by means of virtual energy tanks, a control technique developed to achieve closed-loop passivity for any arbitrary control input. Albeit the latter result is heavily used, we discuss why its practical application at its current stage remains rather limited, which makes contact with the highly debated claim that passivity-based techniques are associated with a loss of performance. The use of RL allows us to learn a control policy that can be passivized using the energy tank architecture, combining the versatility of learning approaches and the system theoretic properties which can be inferred due to the energy tanks. Simulations show the validity of the approach, as well as novel interesting research directions in energy-aware robotics.
Paper Structure (13 sections, 1 theorem, 17 equations, 5 figures, 1 table)

This paper contains 13 sections, 1 theorem, 17 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Passivity-based control and energy tanks
  4. Reinforcement learning
  5. Energy tanks meet RL
  6. Passivization in Inference (Passivizing learned policies)
  7. Passivization in Training (Learning passive policies)
  8. Simulations
  9. DoorOpening
  10. Pendulum
  11. Passivization in Inference
  12. Passivization in Training
  13. Conclusions and Future Work

Key Result

Theorem 1

Assuming constant generalized forces $w_k$ in the sampling interval $[Tk,T(k+1)]$, the PBC objective (eq:passivityEnvDisc) is achieved using the tank update rule (eq:deltaenergy).

Figures (5)

  • Figure 1: The essential of the energy tank algorithm.
  • Figure 2: RL with energy tank passivization
  • Figure 3: DoorOpening environment
  • Figure 4: Average energy spent in DoorOpening in inference.
  • Figure 5: In (a) and (b) respectively the average returns obtained and the energy spent in training. In (c) and (d) respectively the average position error and energy spent in inference.

Theorems & Definitions (2)

  • Theorem
  • proof