Strategically Linked Decisions in Long-Term Planning and Reinforcement Learning

TL;DR

This paper introduces strategic link scores, defined as the drop in the likelihood of a setup decision when a necessary pay-off decision is blocked, formalized as $\mathfrak{S}^{\dagger}_{(s,a)\to(\tilde{s},\tilde{a})} = \pi^{\dagger}(a|s) - \pi^{\dagger:\{\pi(\tilde{a}|\tilde{s})=0\}}(a|s)$, to capture interdependencies in long-horizon planning. It demonstrates three main applications: planning-level explanations for RL by identifying strategically linked decisions, safe policy improvement through strategy-aware recommendations, and interventions-based characterization of planning behavior in both RL and non-RL agents (including a traffic routing scenario). The approach is illustrated with GridWorld experiments, a Shortcuts environment, and a realistic traffic simulator, showing that strategic links can be inferred from demonstrations via inverse reinforcement learning or from known planners. The results highlight improved explainability, safer incremental improvements, and the ability to quantify planning horizons through interventions, offering a broadly applicable tool for understanding and guiding strategic decision-making under constraints.

Abstract

Long-term planning, as in reinforcement learning (RL), involves finding strategies: actions that collectively work toward a goal rather than individually optimizing their immediate outcomes. As part of a strategy, some actions are taken at the expense of short-term benefit to enable future actions with even greater returns. These actions are only advantageous if followed up by the actions they facilitate, consequently, they would not have been taken if those follow-ups were not available. In this paper, we quantify such dependencies between planned actions with strategic link scores: the drop in the likelihood of one decision under the constraint that a follow-up decision is no longer available. We demonstrate the utility of strategic link scores through three practical applications: (i) explaining black-box RL agents by identifying strategically linked pairs among decisions they make, (ii) improving the worst-case performance of decision support systems by distinguishing whether recommended actions can be adopted as standalone improvements or whether they are strategically linked hence requiring a commitment to a broader strategy to be effective, and (iii) characterizing the planning processes of non-RL agents purely through interventions aimed at measuring strategic link scores - as an example, we consider a realistic traffic simulator and analyze through road closures the effective planning horizon of the emergent routing behavior of many drivers.

Figures (12)

• Figure 1: An example strategy in a navigation task. Picking up the key takes extra time early on but unlocks a major shortcut later. If the shortcut were to be blocked, going for the key would no longer be optimal, showing that picking up the key and taking the shortcut are strategically linked---the key is picked up specifically to be able to take the shortcut.
• Figure 2: Dynamics of the toy example. For initial state $S_1$, the action sequence $(A_2,A_2)$ is optimal under both reward functions ${r_{\alpha}}$ and ${r_{\beta}}$. For ${r_{\alpha}}$, the two actions are strategically linked as $A_2|S_1$ is not optimal unless $A_2|S_2$. For ${r_{\beta}}$, there is no strategic link as $A_2|S_1$ is optimal regardless of the action taken at state $S_2$.
• Figure 3: Strategic link scores for a simple maze layout. Creating a shortcut by picking up the key to unlocking the door is strategic (a), since blocking the door results in the key not being picked up (b), while constraining an unrelated action does not lead to the same outcome (c). By looking at the strategic link scores between all the decisions along the optimal trajectory (d), the link between the key ("K") and the door ("D") can be seen clearly.
• Figure 4: Strategic link scores for layout with independent vs. correlated keys. When the keys unlock separate shortcuts (a), they is no strategically link between them (b). If one key is skipped, collecting the other still remains optimal. When the keys jointly unlock a single shortcut (c), the decisions to pick up each key are strategically linked (d). If one key is skipped, collecting the other becomes pointless.
• Figure 5: Strategic link scores inferred from demonstrations become more accurate with increasing variation, following a similar trend to reward inference (until policies are almost uniformly random and rewards become unidentifiable, strategic link scores remain accurate as recognizing policies to be uniformly random is sufficient).