Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Object Permanence Filter for Robust Tracking with Interactive Robots

Shaoting Peng, Margaret X. Wang, Julie A. Shah, Nadia Figueroa

TL;DR

The paper tackles robust tracking in human-robot interaction under unreliable perception and occlusion by introducing the Object Permanence Filter (OPF), which embeds object permanence into a particle filter. The OPF augments the PF with an Object Permanence Update (OP Update) that combines a Dynamics Module, Occluder Module, and Uncertainty Module to generate virtual measurements and adapt covariance during occlusion, plus a Feedback Module to manage safety. A Closed-loop Tracking Controller uses the uncertainty signal to modulate gains, enabling cautious robot behavior, and the approach is demonstrated through extensive simulations and hardware experiments across multiple measurement types. The results show OPF significantly improves tracking robustness in occlusion-heavy scenarios and can generalize across sensors and tasks, providing a practical pathway toward safer, more resilient interactive robots.

Abstract

Object permanence, which refers to the concept that objects continue to exist even when they are no longer perceivable through the senses, is a crucial aspect of human cognitive development. In this work, we seek to incorporate this understanding into interactive robots by proposing a set of assumptions and rules to represent object permanence in multi-object, multi-agent interactive scenarios. We integrate these rules into the particle filter, resulting in the Object Permanence Filter (OPF). For multi-object scenarios, we propose an ensemble of K interconnected OPFs, where each filter predicts plausible object tracks that are resilient to missing, noisy, and kinematically or dynamically infeasible measurements, thus bringing perceptional robustness. Through several interactive scenarios, we demonstrate that the proposed OPF approach provides robust tracking in human-robot interactive tasks agnostic to measurement type, even in the presence of prolonged and complete occlusion. Webpage: https://opfilter.github.io/.

Object Permanence Filter for Robust Tracking with Interactive Robots

TL;DR

The paper tackles robust tracking in human-robot interaction under unreliable perception and occlusion by introducing the Object Permanence Filter (OPF), which embeds object permanence into a particle filter. The OPF augments the PF with an Object Permanence Update (OP Update) that combines a Dynamics Module, Occluder Module, and Uncertainty Module to generate virtual measurements and adapt covariance during occlusion, plus a Feedback Module to manage safety. A Closed-loop Tracking Controller uses the uncertainty signal to modulate gains, enabling cautious robot behavior, and the approach is demonstrated through extensive simulations and hardware experiments across multiple measurement types. The results show OPF significantly improves tracking robustness in occlusion-heavy scenarios and can generalize across sensors and tasks, providing a practical pathway toward safer, more resilient interactive robots.

Abstract

Object permanence, which refers to the concept that objects continue to exist even when they are no longer perceivable through the senses, is a crucial aspect of human cognitive development. In this work, we seek to incorporate this understanding into interactive robots by proposing a set of assumptions and rules to represent object permanence in multi-object, multi-agent interactive scenarios. We integrate these rules into the particle filter, resulting in the Object Permanence Filter (OPF). For multi-object scenarios, we propose an ensemble of K interconnected OPFs, where each filter predicts plausible object tracks that are resilient to missing, noisy, and kinematically or dynamically infeasible measurements, thus bringing perceptional robustness. Through several interactive scenarios, we demonstrate that the proposed OPF approach provides robust tracking in human-robot interactive tasks agnostic to measurement type, even in the presence of prolonged and complete occlusion. Webpage: https://opfilter.github.io/.
Paper Structure (21 sections, 12 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 21 sections, 12 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. OPF: Object Permanence Filter
  4. PF for Object Tracking
  5. Object Permanence Update
  6. Dynamics Module
  7. Occluder Module
  8. Uncertainty Module
  9. Feedback Module
  10. Human Intervention Feedback
  11. Closed-loop Tracking Controller
  12. Experiments and Results
  13. Metrics and Evaluation Protocol
  14. Comparative Simulation Experiments
  15. General OP Tracking
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Our Object Permanence Filter can be used for robust tracking of occluded objects or noisy measurements with different tracking systems at different frequencies. Apriltags (30Hz) used on the left for the cups game and sugar-dropping experiment and Optitrack (100Hz) used on the right for tracking a flying object.
  • Figure 2: Object Permanence Filter (OPF): Following the predict-update cycle, the OPF introduces OP update when no measurements are available for the object being tracked. OP update consists of i) dynamics module to detect and model the dynamics of the object, ii) occluder module to help decide the occluder and deal with multiple occluders, and iii) uncertainty module to update the covariance matrix. The feedback module monitors the uncertainty of the updates by tracking the trace of the update covariance matrix, which can be used to change the behavior of the robot or indicate to the human operator when the uncertainty is above a safety threshold $\epsilon_{safe}$.
  • Figure 3: Comparative Results: (top row) General object permanence (OP) tracking experiment, (bottom row) sugar-dropping experiment (inspired by maskukf). The X-axis for all plots denotes $k$-th camera frame. (1st column) simulation snapshots, (2nd-3rd columns) tracking error distances of translation and rotation of the occluded object given by PF (blue) and OPF (red) (4th column) traces of $Q$ (Eq. \ref{['eq:pf_update']}).