Beyond Quadratic Costs in LQR: Bregman Divergence Control

TL;DR

This paper extends stochastic optimal control beyond quadratic costs by formulating a Bregman-divergence-based framework. By enforcing a quadratic value function in the state, the approach decouples deterministic cost dynamics from stochastic disturbances, allowing closed-form expectations and a generalized Riccati-like equation that supports nonlinear state feedback. The resulting Bregman controllers guarantee stability and accommodate diverse penalties, including elastic-net (l1+l2) terms, yielding sparse or bounded behavior and enabling applications such as safety and sparse control. The framework provides two design paths—specifying either the state cost q or the control cost r—while ensuring the other is compatible via convex feasibility conditions. Collectively, the work broadens the spectrum of tractable, robust control strategies for infinite-horizon, noisy environments with non-quadratic convex costs.

Abstract

In the past couple of decades, the use of ``non-quadratic" convex cost functions has revolutionized signal processing, machine learning, and statistics, allowing one to customize solutions to have desired structures and properties. However, the situation is not the same in control where the use of quadratic costs still dominates, ostensibly because determining the ``value function", i.e., the optimal expected cost-to-go, which is critical to the construction of the optimal controller, becomes computationally intractable as soon as one considers general convex costs. As a result, practitioners often resort to heuristics and approximations, such as model predictive control that only looks a few steps into the future. In the quadratic case, the value function is easily determined by solving Riccati equations. In this work, we consider a special class of convex cost functions constructed from Bregman divergence and show how, with appropriate choices, they can be used to fully extend the framework developed for the quadratic case. The resulting optimal controllers are infinite horizon, come with stability guarantees, and have state-feedback, or estimated state-feedback, laws. They exhibit a much wider range of behavior than their quadratic counterparts since the feedback laws are nonlinear. The approach can be applied to several cases of interest, including safety control, sparse control, and bang-bang control.

Figures (5)

• Figure 1: State trajectories, control inputs and costs with the bang-bang control cost, for a scalar system with $t=4$.
• Figure 2: State trajectories, control inputs, and cost functions for an exponential control cost ($m=0.3$).
• Figure 3: State trajectories and control inputs with the elastic-net state cost for $m=0.01$.
• Figure 4: State trajectories and control inputs with the elastic-net state cost for $m=0.07$.
• Figure 5: Comparison of the $q$ and $r$ functions for different values of $m$.

Theorems & Definitions (10)

• Example 1
• Example 2
• Lemma 1
• Lemma 2
• Remark 1
• Theorem 1: Optimal Controller Synthesis via KKT Conditions
• Theorem 2
• Theorem 3
• Lemma 3
• Lemma 4