RECON: Reducing Causal Confusion with Human-Placed Markers

TL;DR

The paper tackles causal confusion in imitation learning caused by extraneous visual cues by enabling humans to place beacons on task‑relevant objects. It introduces RECON, which learns a task‑relevant feature $φ = f_{θ_2}(x,y)$ via a feature network, uses a policy network $π_{θ_1}$ to map $(x,φ)$ to actions, and employs a beacon decoder $h_{θ_3}$ to relate $φ$ to beacon readings $b$ during training; beacons are removed during execution and do not constrain the policy. The training objective combines action imitation and beacon reconstruction via $L = L_1 + L_2$, encouraging $φ$ to align with beacon signals while preserving task performance. Simulations and a real UR5 robot demonstrate that RECON reduces the number of demonstrations required and improves task accuracy, even under imperfect beacon data or post‑training beacon removal. Overall, the approach provides a practical way to inject human knowledge into visual imitation learning to robustly mitigate causal confusion in robotic tasks.

Abstract

Imitation learning enables robots to learn new tasks from human examples. One fundamental limitation while learning from humans is causal confusion. Causal confusion occurs when the robot's observations include both task-relevant and extraneous information: for instance, a robot's camera might see not only the intended goal, but also clutter and changes in lighting within its environment. Because the robot does not know which aspects of its observations are important a priori, it often misinterprets the human's examples and fails to learn the desired task. To address this issue, we highlight that -- while the robot learner may not know what to focus on -- the human teacher does. In this paper we propose that the human proactively marks key parts of their task with small, lightweight beacons. Under our framework (RECON) the human attaches these beacons to task-relevant objects before providing demonstrations: as the human shows examples of the task, beacons track the position of marked objects. We then harness this offline beacon data to train a task-relevant state embedding. Specifically, we embed the robot's observations to a latent state that is correlated with the measured beacon readings: in practice, this causes the robot to autonomously filter out extraneous observations and make decisions based on features learned from the beacon data. Our simulations and a real robot experiment suggest that this framework for human-placed beacons mitigates causal confusion. Indeed, we find that using RECON significantly reduces the number of demonstrations needed to convey the task, lowering the overall time required for human teaching. See videos here: https://youtu.be/oy85xJvtLSU

Figures (5)

• Figure 1: (Left) Human teaching a robot arm to place bread on a blue plate. The environment also includes a red bowl that is not relevant to the task. The robot observes this scene from a top-down camera. We enable humans to mark task-relevant objects (i.e., the plate) with beacons --- lightweight devices that track the position of the marked objects while the human is demonstrating the task. (Center) A robot trained to imitate the task with only visual information incorrectly infers that both objects are relevant and delivers the bread to the wrong location. (Right) A robot trained with beacons using our RECON algorithm is able to mitigate this confusion and learns to correctly drop the bread onto the plate, even after the beacons are removed.
• Figure 2: Our proposed model architecture consists of a feature network that maps observations $(x, y)$ to task-relevant features $\phi$, a beacon network that relates the features to beacon data $b$, and a policy network that estimates actions $u$ based on the robot state and learned features. The beacon network is only utilized at training time to supervise the task-relevant features.
• Figure 3: Environments and results for our simulations in Section \ref{['sec:sim1']} averaged over $20$ training and testing runs. (Left) In the Static 2D environment, the robot must reach the object in the center. Here a robot trained with position beacons (RECON-P) reaches closer to the target than a robot trained without beacons (Baseline). However, when using distance beacons (RECON-D) we require additional Play data (i.e., state and beacon pairs) to properly calibrate the feature network and ensure that the robot performs better than the baseline. (Right) In the Robosuite environment, the robot must transfer the right-most object from one bin to the other. Here the robot imitates the expert policy more closely when trained with position beacons and play data (RECON-P (Play)), while training with distance beacons and play data (RECON-D (Play)) produces results similar to the Baseline.
• Figure 4: Simulation results averaged over $15$ training and testing runs in the Dynamic 2D environment. We find that robots trained with beacons attached to the task-relevant objects (Exact) achieve the highest rewards while random beacon placement results in the lowest. Though Partial and Other (less relevant) beacon readings also perform worse than Exact, they still outperform the Baseline (which does not use beacons). Exact remains better than the Baseline even with moderate noise ($\sigma=2.5$), only performing worse at high noise levels ($\sigma > 4.5$).
• Figure 5: Results for our robot experiment averaged over $3$ end-to-end runs. With just four demonstrations, a robot trained without beacons (Baseline) fails to recognize the task-relevant objects (blue plate), resulting in poor performance during testing. In contrast, a robot trained with beacon readings (RECON (Play)) learns to focus on the plate and deliver the toast accurately.