Causal Imitation Learning under Temporally Correlated Noise

TL;DR

The paper addresses imitation learning under temporally correlated noise that couples past and current actions, causing standard IL methods like behavioral cloning to fail. It leverages instrumental variable regression with past states as instruments to identify the expert's policy in a confounded setting, presenting two algorithms: DoubIL (simulator-based) and ResiduIL (offline). The authors formalize confounding, derive PRMSE-based objectives and corresponding performance bounds, and validate the methods on multiple simulated control tasks where they outperform behavioral cloning and closely match the expert under confounding. The work provides tools for robust offline imitation learning in sequential settings and offers a practical path to detect and mitigate causal confounding in real data.

Abstract

We develop algorithms for imitation learning from policy data that was corrupted by temporally correlated noise in expert actions. When noise affects multiple timesteps of recorded data, it can manifest as spurious correlations between states and actions that a learner might latch on to, leading to poor policy performance. To break up these spurious correlations, we apply modern variants of the instrumental variable regression (IVR) technique of econometrics, enabling us to recover the underlying policy without requiring access to an interactive expert. In particular, we present two techniques, one of a generative-modeling flavor (DoubIL) that can utilize access to a simulator, and one of a game-theoretic flavor (ResiduIL) that can be run entirely offline. We find both of our algorithms compare favorably to behavioral cloning on simulated control tasks.

Figures (7)

• Figure 1: (a) When temporally correlated noise (e.g. wind) affects expert actions, standard imitation learning approaches like behavioral cloning can amplify this noise, leading to poor test-time performance. (b) TCN $u_t$ affects both the input ($s_t$) and output ($a_t$) of our learning procedure. This breaks a cardinal assumption of regression-based approaches like behavioral cloning, rendering them inconsistent. (c) We can re-simulate state transitions from a past state, producing fresh samples ($\widetilde{s_t}$). We can then regress from these sampled states to observed expert actions to recover the expert's policy as the noise on inputs and outputs is no longer correlated.
• Figure 2: The structural causal model (SCM) considered in IVR. We are interested in finding $h$, the causal relationship from $X$ to $Y$, even though there is an unobserved confounder, $U$. We do so by leveraging the effect of $Z$, which provides randomness independent of $U$.
• Figure 3: An SCM that captures TCN. The confounding ($U = u_{t-1}$) is mediated via the dynamics into the state, introducing spurious correlations between states ($X = s_t$) and actions ($Y = a_t$). To break the confounding, we can utilize the past state as an instrument ($Z = s_{t-1}$).
• Figure 4: DoubIL deconfounds inputs to the second stage regression by re-sampling state transitions via simulator $\widehat{\mathcal{T}}$.
• Figure 5: We train behavioral cloning, DoubIL, and ResiduIL on trajectories from a modified LunarLander environment, computing standard errors across four runs. Left:DoubIL and ResiduIL are better able to match $\pi_E(s) = \mathbb{E}[a|do(s)]$ on states from expert rollouts. Center: The policies learned by our algorithms generalize better than those produced by behavioral cloning to the state distribution of the expert on the noiseless problem ($u_t = 0$). Right: We can compare the results of behavioral cloning to one of our causal IL procedures to identify areas of the state space where the effect of confounding is strong (the red dots).

Theorems & Definitions (12)

• Theorem 3.1
• Theorem 3.2
• Theorem 5.1
• Lemma 5.2
• Lemma 5.3
• Lemma 6.1
• proof
• proof
• proof
• proof