Beyond Domain Randomization: Event-Inspired Perception for Visually Robust Adversarial Imitation from Videos

TL;DR

The paper addresses visual domain gaps in imitation from observations by eliminating appearance cues through an event-inspired perception that converts RGB video into a sparse temporal-gradient representation. It introduces EB-LAIfO, which performs latent adversarial imitation in the event space, enabling robust imitation despite appearance mismatches and reducing the need for costly data augmentation. Empirical results on the DeepMind Control Suite and Adroit demonstrate superior robustness and sample efficiency against strong baselines, including near-expert performance in several settings. The work highlights practical implications for real-world deployment and suggests future extensions with real event cameras and richer event-based representations.

Abstract

Imitation from videos often fails when expert demonstrations and learner environments exhibit domain shifts, such as discrepancies in lighting, color, or texture. While visual randomization partially addresses this problem by augmenting training data, it remains computationally intensive and inherently reactive, struggling with unseen scenarios. We propose a different approach: instead of randomizing appearances, we eliminate their influence entirely by rethinking the sensory representation itself. Inspired by biological vision systems that prioritize temporal transients (e.g., retinal ganglion cells) and by recent sensor advancements, we introduce event-inspired perception for visually robust imitation. Our method converts standard RGB videos into a sparse, event-based representation that encodes temporal intensity gradients, discarding static appearance features. This biologically grounded approach disentangles motion dynamics from visual style, enabling robust visual imitation from observations even in the presence of visual mismatches between expert and agent environments. By training policies on event streams, we achieve invariance to appearance-based distractors without requiring computationally expensive and environment-specific data augmentation techniques. Experiments across the DeepMind Control Suite and the Adroit platform for dynamic dexterous manipulation show the efficacy of our method. Our code is publicly available at Eb-LAIfO.

Figures (7)

• Figure 1: Summary of EB-LAIfO. Given the RGB sequence, the corresponding events stream is extracted following \ref{['eq:event_transf']}. A feature extractor network $\phi_{\delta}$ is used to generate the latent features $z_\delta$ used by both the Q-function $Q_{\bm{\psi}}$ and discriminator $D_\chi$ for the imitation problem. The discriminator $D_\chi$ is trained as in \ref{['eq:AIL_BCE']} and returns the reward function $r_{\bm{\chi}}$ which is then maximized through an RL step. The RL step is described in \ref{['eq:DDPG_critic']} and follows the Deep Deterministic Policy Gradient (DDPG) pipeline silver2014deterministic. Our feature extractor network $\phi_{\delta}$ is trained jointly with the Q-function and extracts goal-relevant information directly from the events stream.
• Figure 2: Examples scenes from the expert and target domain, both in original RGB space and obtained event spaced (green is positive event, red is negative).
• Figure 3: Learning curves for the results in Table \ref{['table_dmc']}. Plots show the average return per episode and the standard deviation across seeds as a function of training steps.
• Figure 4: Learning curves for the results described in Table \ref{['table_adroit']}. Plots show the average return per episode and the standard deviation across seeds as a function of training steps.
• Figure 5: Adroit environments used for the experiments in Table \ref{['table_adroit']}.