Control Synthesis of Cyber-Physical Systems for Real-Time Specifications through Causation-Guided Reinforcement Learning
Xiaochen Tang, Zhenya Zhang, Miaomiao Zhang, Jie An
TL;DR
This work tackles real-time control synthesis for safety-critical CPS under STL specifications by addressing the non-Markovian rewards of STL-guided RL. It introduces online causation-guided RL, leveraging STL online causation semantics and a differentiable reward via tau-MDP to provide informative, instantaneous feedback for deep RL. Empirical results across Cart-Pole, Reach-Avoid, Hopper, and Walker show improved convergence, higher satisfaction metrics (Full-SAT, Safety-SAT), and lower cost returns compared with hand-crafted rewards and prior STL-based methods. The approach offers a scalable framework for robust real-time policy synthesis in CPS, with manageable computational overhead and clear directions for future enhancements like reduced-cost implementations and broader applicability.
Abstract
In real-time and safety-critical cyber-physical systems (CPSs), control synthesis must guarantee that generated policies meet stringent timing and correctness requirements under uncertain and dynamic conditions. Signal temporal logic (STL) has emerged as a powerful formalism of expressing real-time constraints, with its semantics enabling quantitative assessment of system behavior. Meanwhile, reinforcement learning (RL) has become an important method for solving control synthesis problems in unknown environments. Recent studies incorporate STL-based reward functions into RL to automatically synthesize control policies. However, the automatically inferred rewards obtained by these methods represent the global assessment of a whole or partial path but do not accumulate the rewards of local changes accurately, so the sparse global rewards may lead to non-convergence and unstable training performances. In this paper, we propose an online reward generation method guided by the online causation monitoring of STL. Our approach continuously monitors system behavior against an STL specification at each control step, computing the quantitative distance toward satisfaction or violation and thereby producing rewards that reflect instantaneous state dynamics. Additionally, we provide a smooth approximation of the causation semantics to overcome the discontinuity of the causation semantics and make it differentiable for using deep-RL methods. We have implemented a prototype tool and evaluated it in the Gym environment on a variety of continuously controlled benchmarks. Experimental results show that our proposed STL-guided RL method with online causation semantics outperforms existing relevant STL-guided RL methods, providing a more robust and efficient reward generation framework for deep-RL.