Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Discrete fully probabilistic design: towards a control pipeline for the synthesis of policies from examples

Enrico Ferrentino, Pasquale Chiacchio, Giovanni Russo

TL;DR

The paper addresses learning control policies from example data for constrained, stochastic, nonlinear systems, allowing data from a different system and without requiring constraints to hold in the examples. It introduces a discrete fully probabilistic design (DFPD) framework that uses Bayesian probability models and KL-divergence minimization between the realized behavior $P^n$ and a reference $Q^n$, accompanied by a practical pipeline to convert data into discrete probability inputs for control. The authors present a backward-recursive DFPD algorithm that computes an optimal probabilistic controller and demonstrate its viability by benchmarking on an inverted pendulum with actuation constraints, using data gathered from a physically different pendulum. Open-source, documented code accompanies the work, and the authors discuss extensions toward an end-to-end demonstration pipeline and techniques for discretization impact analysis and dimensionality reduction to scale the approach.

Abstract

We present the principled design of a control pipeline for the synthesis of policies from examples data. The pipeline, based on a discretized design which we term as discrete fully probabilistic design, expounds an algorithm recently introduced in Gagliardi and Russo (2021) to synthesize policies from examples for constrained, stochastic and nonlinear systems. Contrary to other approaches, the pipeline we present: (i) does not need the constraints to be fulfilled in the possibly noisy example data; (ii) enables control synthesis even when the data are collected from an example system that is different from the one under control. The design is benchmarked numerically on an example that involves controlling an inverted pendulum with actuation constraints starting from data collected from a physically different pendulum that does not satisfy the system-specific actuation constraints. We also make our fully documented code openly available.

Discrete fully probabilistic design: towards a control pipeline for the synthesis of policies from examples

TL;DR

The paper addresses learning control policies from example data for constrained, stochastic, nonlinear systems, allowing data from a different system and without requiring constraints to hold in the examples. It introduces a discrete fully probabilistic design (DFPD) framework that uses Bayesian probability models and KL-divergence minimization between the realized behavior and a reference , accompanied by a practical pipeline to convert data into discrete probability inputs for control. The authors present a backward-recursive DFPD algorithm that computes an optimal probabilistic controller and demonstrate its viability by benchmarking on an inverted pendulum with actuation constraints, using data gathered from a physically different pendulum. Open-source, documented code accompanies the work, and the authors discuss extensions toward an end-to-end demonstration pipeline and techniques for discretization impact analysis and dimensionality reduction to scale the approach.

Abstract

We present the principled design of a control pipeline for the synthesis of policies from examples data. The pipeline, based on a discretized design which we term as discrete fully probabilistic design, expounds an algorithm recently introduced in Gagliardi and Russo (2021) to synthesize policies from examples for constrained, stochastic and nonlinear systems. Contrary to other approaches, the pipeline we present: (i) does not need the constraints to be fulfilled in the possibly noisy example data; (ii) enables control synthesis even when the data are collected from an example system that is different from the one under control. The design is benchmarked numerically on an example that involves controlling an inverted pendulum with actuation constraints starting from data collected from a physically different pendulum that does not satisfy the system-specific actuation constraints. We also make our fully documented code openly available.
Paper Structure (16 sections, 1 theorem, 16 equations, 4 figures, 1 algorithm)

This paper contains 16 sections, 1 theorem, 16 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Mathematical Set-up and Problem Statement
  3. Formulation of the control problem
  4. The Discrete Fully Probabilistic Design Algorithm
  5. The Proposed Design Pipeline: From Data to Control Inputs
  6. Data gathering
  7. Quantization
  8. Computation of the probability functions
  9. Respecting the range of feasible inputs
  10. Adding constraints in the optimization problem
  11. Benchmarking the DFPD on an Inverted Pendulum
  12. The environment
  13. Obtaining the inputs to Algorithm \ref{['alg:discrete_fpd']}.
  14. Results
  15. Conclusions
  16. ...and 1 more sections

Key Result

Proposition 3.2

Algorithm alg:discrete_fpd returns the optimal solution to Problem prb:global_opt.

Figures (4)

  • Figure 1: Reference system data (single trajectories) over 100 simulations.
  • Figure 2: Reference system data over 100 simulations (same as Figure \ref{['fig:reference_data_single']}); bold lines denote means, shaded areas the confidence intervals corresponding to the standard deviation.
  • Figure 3: Time evolution of position, velocity and torque when the data-driven control policy is used to control the target (blue) and reference (red) system. Colors online.
  • Figure 4: Target pendulum trajectory through data-driven control with restricted torque constraints

Theorems & Definitions (8)

  • Proposition 3.2
  • proof
  • Remark 3.3
  • Remark 3.4
  • Remark 4.1
  • Remark 4.2
  • Remark 4.3
  • Remark 4.4