Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning

TL;DR

This work tackles the difficulty of learning goal-conditioned policies when rewards are sparse in object-centric domains. It introduces Null Counterfactual Interaction Inference (NCII) to identify object interactions via null counterfactuals and Hindsight Relabeling using Interactions (HInt) to filter hindsight data, aligning the replay distribution with meaningful interactions. NCII trains a masked forward dynamics model and an interaction predictor to detect interacting state factors, then HInt uses these interactions to select hindsight trajectories that reflect actual control over target objects, yielding up to 4x gains in sample efficiency across diverse robotics domains. Together, NCII and HInt provide a generalizable inductive bias for GCRL, improving data efficiency and robustness in environments where interactions drive goal achievement.

Abstract

Hindsight relabeling is a powerful tool for overcoming sparsity in goal-conditioned reinforcement learning (GCRL), especially in certain domains such as navigation and locomotion. However, hindsight relabeling can struggle in object-centric domains. For example, suppose that the goal space consists of a robotic arm pushing a particular target block to a goal location. In this case, hindsight relabeling will give high rewards to any trajectory that does not interact with the block. However, these behaviors are only useful when the object is already at the goal -- an extremely rare case in practice. A dataset dominated by these kinds of trajectories can complicate learning and lead to failures. In object-centric domains, one key intuition is that meaningful trajectories are often characterized by object-object interactions such as pushing the block with the gripper. To leverage this intuition, we introduce Hindsight Relabeling using Interactions (HInt), which combines interactions with hindsight relabeling to improve the sample efficiency of downstream RL. However because interactions do not have a consensus statistical definition tractable for downstream GCRL, we propose a definition of interactions based on the concept of null counterfactual: a cause object is interacting with a target object if, in a world where the cause object did not exist, the target object would have different transition dynamics. We leverage this definition to infer interactions in Null Counterfactual Interaction Inference (NCII), which uses a "nulling'' operation with a learned model to infer interactions. NCII is able to achieve significantly improved interaction inference accuracy in both simple linear dynamics domains and dynamic robotic domains in Robosuite, Robot Air Hockey, and Franka Kitchen and HInt improves sample efficiency by up to 4x.

Figures (16)

• Figure 1: Figure (a) shows a case when a null counterfactual interaction occurs between the cause object and the target object, by comparing the actual event (left) with the null counterfactual ("nulled") event (right) and observing a difference in the target velocity. Figure (b) shows when an interaction does not occur since the actual event matches the null counterfactual for the target object.
• Figure 2: An example of the unrolled dynamic interaction graph, where an edge indicates an interaction $\mathbb B_{ji}^{(t)} = 1$ from $t$ to $t+1$. HInt identifies trajectories where the agent exerts control on the target object, as measured by a path in the unrolled interaction graph. The colors indicate the timesteps during which each state factor is controlled: $S_1$ is controlled from $t=2$ to $t=6$, $S_2$ is not controlled, and $S_\text{tar}$ is controlled from $t=4$ to $t=6$.
• Figure 3: Visualizations of domains used for evaluation. Goals are in green and target objects are in red (except for Franka Kitchen domain).
• Figure 4: Comparison of HInt and HInt with NCII against baselines, 5 trials for each. HInt with NCII is used in Spriteworld default, small, Robosuite default, Air Hockey default, Kitchen default, and obstacles. The shading indicates standard error. X axis is the number of timesteps.
• Figure 5: Relative position heatmap between initial state and a) sampled or "desired" goal, b) hindsight goals, c) goals after HInt filtering, d) goals removed by HInt, over 3000 goals in Spriteworld default.

Theorems & Definitions (3)

• Definition 3.1: Null Counterfactual Interaction Assumption
• Definition I.1: Null Actual Cause Necessity
• Definition I.2: Null Actual Cause