Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Actron3D: Learning Actionable Neural Functions from Videos for Transferable Robotic Manipulation

Anran Zhang, Hanzhi Chen, Yannick Burkhardt, Yao Zhong, Johannes Betz, Helen Oleynikova, Stefan Leutenegger

TL;DR

Actron3D tackles learning transferable 6-DoF robotic manipulation from a few monocular RGB videos by distilling them into Neural Affordance Functions (NAFs), compact object-centric neural representations that encode geometry, appearance, affordances, and action flows. A memory bank of NAFs is retrieved via multimodal reasoning with an MLLM, then transferred to a new scene through a coarse-to-fine differentiable SIM(3) alignment coupled with affordance optimization to yield executable 6-DoF policies. The approach demonstrates substantial gains over state-of-the-art baselines in simulation and real-robot experiments using only 2–3 demonstrations per task, and supports downstream tasks such as long-horizon planning and online policy fine-tuning. This framework reduces data requirements, handles viewpoint and object variation via 3D affordance representations, and provides a scalable pathway for embodied generalization in uncalibrated, unconstrained settings.

Abstract

We present Actron3D, a framework that enables robots to acquire transferable 6-DoF manipulation skills from just a few monocular, uncalibrated, RGB-only human videos. At its core lies the Neural Affordance Function, a compact object-centric representation that distills actionable cues from diverse uncalibrated videos-geometry, visual appearance, and affordance-into a lightweight neural network, forming a memory bank of manipulation skills. During deployment, we adopt a pipeline that retrieves relevant affordance functions and transfers precise 6-DoF manipulation policies via coarse-to-fine optimization, enabled by continuous queries to the multimodal features encoded in the neural functions. Experiments in both simulation and the real world demonstrate that Actron3D significantly outperforms prior methods, achieving a 14.9 percentage point improvement in average success rate across 13 tasks while requiring only 2-3 demonstration videos per task.

Actron3D: Learning Actionable Neural Functions from Videos for Transferable Robotic Manipulation

TL;DR

Actron3D tackles learning transferable 6-DoF robotic manipulation from a few monocular RGB videos by distilling them into Neural Affordance Functions (NAFs), compact object-centric neural representations that encode geometry, appearance, affordances, and action flows. A memory bank of NAFs is retrieved via multimodal reasoning with an MLLM, then transferred to a new scene through a coarse-to-fine differentiable SIM(3) alignment coupled with affordance optimization to yield executable 6-DoF policies. The approach demonstrates substantial gains over state-of-the-art baselines in simulation and real-robot experiments using only 2–3 demonstrations per task, and supports downstream tasks such as long-horizon planning and online policy fine-tuning. This framework reduces data requirements, handles viewpoint and object variation via 3D affordance representations, and provides a scalable pathway for embodied generalization in uncalibrated, unconstrained settings.

Abstract

We present Actron3D, a framework that enables robots to acquire transferable 6-DoF manipulation skills from just a few monocular, uncalibrated, RGB-only human videos. At its core lies the Neural Affordance Function, a compact object-centric representation that distills actionable cues from diverse uncalibrated videos-geometry, visual appearance, and affordance-into a lightweight neural network, forming a memory bank of manipulation skills. During deployment, we adopt a pipeline that retrieves relevant affordance functions and transfers precise 6-DoF manipulation policies via coarse-to-fine optimization, enabled by continuous queries to the multimodal features encoded in the neural functions. Experiments in both simulation and the real world demonstrate that Actron3D significantly outperforms prior methods, achieving a 14.9 percentage point improvement in average success rate across 13 tasks while requiring only 2-3 demonstration videos per task.
Paper Structure (14 sections, 8 equations, 4 figures, 3 tables)

This paper contains 14 sections, 8 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Problem Formulation
  5. Preliminaries
  6. Neural Functions Distillation from Videos
  7. Affordance Transfer via Neural Functions
  8. Implementation Details
  9. Experiments
  10. Experiment Setup
  11. Results and Discussions
  12. Ablation Studies
  13. Downstream Applications
  14. Conclusion

Figures (4)

  • Figure 1: Overview of Actron3D framework. In the distillation phase, multimodal knowledge from RGB videos is encoded into Neural Affordance Functions, and at deployment, an optimization-based transfer module enables action generalization to novel scenes.
  • Figure 2: Multimodal information and robot actions synthesized from novel views of NAFs fit to diverse videos.
  • Figure 3: Inferred robot gripper actions by different methods.
  • Figure 4: Loss curves of the action head when trained from scratch vs. initialized with pretrained weights of the retrieved NAF model.