Heuristic Search for Linear Positive Systems

TL;DR

This work proposes a heuristics-based algorithm for efficiently finding local solutions to the analyzed class of optimal control problems with a given initial state and positive linear dynamics, and derives a novel distributed algorithm for calculating local controllers within a specified performance bound.

Abstract

This work considers infinite-horizon optimal control of positive linear systems applied to the case of network routing problems. We demonstrate the equivalence between Stochastic Shortest Path (SSP) problems and optimal control of a certain class of linear systems. This is used to construct a heuristic search framework for linear {positive} systems inspired by existing methods for SSP. {We propose a heuristics-based algorithm for {efficiently} finding local solutions to the analyzed class of optimal control problems with {a given initial} state and {positive} linear dynamics.} {By leveraging the bound on optimality in each state provided by the heuristics, we also derive a novel distributed algorithm for calculating local controllers within a specified performance bound, with a distributed condition for termination.} More fundamentally, the results allow for analysis of the conditions for explicit solutions to the Bellman equation utilized by heuristic search methods.

Figures (8)

• Figure 1: Allowed inputs under the constraints of \ref{['eq:optprob']} for the case $m_i = 2$. The objective $(r_i+B_i^\top p)u_i$ is linear in the input $u_i$, so the optimum is attained on at least one of the vertices, for which an explicit expression is available as a linear function of the state $x$. A discrete action set $\mathcal{A}(i)$ can be constructed from the corresponding inputs.
• Figure 2: Expansion of the state space as described in Section 3.3 to accommodate the unit sum transitions of Definition \ref{['def:SSP']}, with multiple states $v_k$ corresponding to the original $v$. A finite SSP cannot represent the unbounded growth in the state that is possible for, e.g., an unstable system in the formulation \ref{['eq:optprob']}. We must instead resort to an infinite space of artificial states with nearly identical properties. As is detailed in wrobel84skeleton, there exist tractable methods for simplification of systems with this structure.
• Figure 3: Illustration of the search space $S$ in Algorithm \ref{['alg:1']}. Optimal cost functions $\overline{g}_S$ and $\underline{g}_S$ with regard to the control $K_S$ are found using $\overline{h}$ and $\underline{h}$ for states outside $S$. The space is expanded in each iteration to include states with high uncertainty $\overline{h}_j-\underline{h}_j$ that are most affected under the optimal control laws $\overline{K}_S$ and $\underline{K}_S$ corresponding to $\overline{g}_S$ and $\underline{g}_S$.
• Figure 4: Illustration of the search space after the first (left) and ninth (right) iteration of Algorithm \ref{['alg:1']} for the system in Example \ref{['ex:big']}. Above, states in dark blue are part of the search space $S$ while shaded states are in the active boundary of states that can be directly affected by the dynamics inside $S$. The red circle indicates the state selected for inclusion in the next iteration. The accompanying plots show the optimal cost function $p$ (dashed red) for each state, as well as the current estimates $\overline{g}$ and $\underline{g}$ (yellow) for states in $S$ and the heuristic bounds $\overline{h}$ and $\underline{h}$ (blue). It is clear from the schematic view of the state space that the algorithm systematically explores states most impacted from the initial configuration. Note in particular that the performance of $x_3$ is quickly controlled to within the performance bound, while $x_2$ remains uncertain, necessitating the exploration of a large part of the remaining state space.
• Figure 5: Upper and lower bound of the total cost in Example \ref{['ex:big']}. The red lines highlight the iterations for which the state is displayed in more detail in Figure \ref{['fig:searchplot']}. The value $\gamma$ specifies the stopping condition in Algorithm \ref{['alg:1']} and guarantees a performance of at least $\overline{g}^\top x_0 \le \gamma p^\top x_0$ when the algorithm terminates. Since the initial heuristics in Example \ref{['ex:big']} are consistent, the convergence is monotone.

Theorems & Definitions (19)

• Remark 1
• Remark 2
• Definition 1: Stochastic Shortest Path Problem
• Lemma 1
• Example 1
• Proposition 2
• Example 2
• Definition 2
• Proposition 3
• Proposition 4