Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inferring Causal Graph Temporal Logic Formulas to Expedite Reinforcement Learning in Temporally Extended Tasks

Hadi Partovi Aria, Zhe Xu

TL;DR

The paper tackles sample inefficiency and interpretability of reinforcement learning on networks with spatial-temporal dynamics. It introduces GTL-CIRL, a closed-loop framework that jointly learns policies and Causal Graph Temporal Logic specifications, using counterexample traces and Gaussian Process-based Bayesian optimization to refine cause templates. Robustness-based rewards and a history-aware, finite-horizon MDP enable principled learning and theoretical guarantees on convergence and bounded regret. Experiments in gene regulation networks and power grids demonstrate faster convergence and clearer, verifiable behavior than standard RL baselines.

Abstract

Decision-making tasks often unfold on graphs with spatial-temporal dynamics. Black-box reinforcement learning often overlooks how local changes spread through network structure, limiting sample efficiency and interpretability. We present GTL-CIRL, a closed-loop framework that simultaneously learns policies and mines Causal Graph Temporal Logic (Causal GTL) specifications. The method shapes rewards with robustness, collects counterexamples when effects fail, and uses Gaussian Process (GP) driven Bayesian optimization to refine parameterized cause templates. The GP models capture spatial and temporal correlations in the system dynamics, enabling efficient exploration of complex parameter spaces. Case studies in gene and power networks show faster learning and clearer, verifiable behavior compared to standard RL baselines.

Inferring Causal Graph Temporal Logic Formulas to Expedite Reinforcement Learning in Temporally Extended Tasks

TL;DR

The paper tackles sample inefficiency and interpretability of reinforcement learning on networks with spatial-temporal dynamics. It introduces GTL-CIRL, a closed-loop framework that jointly learns policies and Causal Graph Temporal Logic specifications, using counterexample traces and Gaussian Process-based Bayesian optimization to refine cause templates. Robustness-based rewards and a history-aware, finite-horizon MDP enable principled learning and theoretical guarantees on convergence and bounded regret. Experiments in gene regulation networks and power grids demonstrate faster convergence and clearer, verifiable behavior than standard RL baselines.

Abstract

Decision-making tasks often unfold on graphs with spatial-temporal dynamics. Black-box reinforcement learning often overlooks how local changes spread through network structure, limiting sample efficiency and interpretability. We present GTL-CIRL, a closed-loop framework that simultaneously learns policies and mines Causal Graph Temporal Logic (Causal GTL) specifications. The method shapes rewards with robustness, collects counterexamples when effects fail, and uses Gaussian Process (GP) driven Bayesian optimization to refine parameterized cause templates. The GP models capture spatial and temporal correlations in the system dynamics, enabling efficient exploration of complex parameter spaces. Case studies in gene and power networks show faster learning and clearer, verifiable behavior compared to standard RL baselines.
Paper Structure (12 sections, 2 theorems, 10 equations, 2 figures, 3 algorithms)

This paper contains 12 sections, 2 theorems, 10 equations, 2 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Q-Learning with GTL Objectives
  4. GTL-CIRL Framework
  5. Counterexample Generation
  6. Theoretical Analysis
  7. Implementation and Experiments
  8. Baseline Methods
  9. Gene Regulation Network
  10. Power Grid Management
  11. Conclusion
  12. Acknowledgments

Key Result

Theorem 1

For finite $\tau$-MDP with GTL reward (Eq. eq:reward) and Robbins-Monro conditions, $Q_k$ converges to $Q^*$ with probability 1.

Figures (2)

  • Figure 1: Graph Temporal Logic for the motivational gene-regulation example. (a) A biological unit $v$ with neighbors $u_1,u_2,w$. (b) Temporal windows encode the ordered edits (cause) and the outcome window (effect).
  • Figure 2: Success rates of GTL-CIRL compared to baseline methods across two domains.

Theorems & Definitions (4)

  • Theorem 1: Convergence of Q-Learning with GTL Robustness Rewards
  • proof : Sketch
  • Theorem 2: Regret Bounds
  • proof : Sketch