Active Measuring in Reinforcement Learning With Delayed Negative Effects

TL;DR

The paper addresses learning in RL when measurements of latent states are costly and can impair the environment. It introduces Actively Observable MDPs (AOMDPs), frames them as a 2-step periodic POMDP, and develops an online belief-state RL algorithm using sequential Monte Carlo to jointly infer latent states and unknown environment parameters. The authors prove polynomial sample complexity for tabular AOMDPs and show that measuring can improve policy value by trading off immediate information gains against delayed environmental effects. Their HeartSteps digital-health experiments demonstrate that active measuring can outperform baselines under realistic conditions, highlighting practical gains in sample efficiency and decision quality. The work provides a concrete, scalable framework for actively acquiring latent-state information in RL with delayed consequences, with implications for healthcare, robotics, and other real-world domains.

Abstract

Measuring states in reinforcement learning (RL) can be costly in real-world settings and may negatively influence future outcomes. We introduce the Actively Observable Markov Decision Process (AOMDP), where an agent not only selects control actions but also decides whether to measure the latent state. The measurement action reveals the true latent state but may have a negative delayed effect on the environment. We show that this reduced uncertainty may provably improve sample efficiency and increase the value of the optimal policy despite these costs. We formulate an AOMDP as a periodic partially observable MDP and propose an online RL algorithm based on belief states. To approximate the belief states, we further propose a sequential Monte Carlo method to jointly approximate the posterior of unknown static environment parameters and unobserved latent states. We evaluate the proposed algorithm in a digital health application, where the agent decides when to deliver digital interventions and when to assess users' health status through surveys.

Figures (6)

• Figure 1: The diagram showing the environment of an AOMDP. A directed edge connected to a square indicates edges to each node within the square. Edges pointing to the control ($A_t$) and measurement actions ($I_t$) are omitted.
• Figure 2: Causal DAG of HeartSteps. Arrows pointing to the actions are omitted.
• Figure 3: The average cumulative reward, subtracting the average cumulative rewards of the zero policy.
• Figure C.1: The measurement rate.
• Figure C.2: The MSE of $\widehat{\bm{\theta}}_t^{R(j)}$.

Theorems & Definitions (11)

• Definition 1: AOMDP
• Lemma 1
• Definition 2: $m$-step $\alpha$-weakly revealing condition
• Proposition 2
• Proposition 3
• Definition 3: $K$-Periodic POMDP
• Definition 4: $K$-Periodic Belief MDP
• proof
• proof
• proof