Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FoAM: Foresight-Augmented Multi-Task Imitation Policy for Robotic Manipulation

Litao Liu, Wentao Wang, Yifan Han, Zhuoli Xie, Pengfei Yi, Junyan Li, Yi Qin, Wenzhao Lian

TL;DR

FoAM addresses reliability and generalization gaps in multi-task imitation learning by introducing a multi-modal goal-conditioned policy and foresight augmentation, enabling the agent to reason about the consequences of its actions. It fuses language instructions with generated goal images from a fine-tuned Vision-Language Model and learns via a CVAE-transformer policy that predicts action chunks and foresight sequences, optimized with action and foresight losses plus KL regularization. Empirical results across over 100 tasks in simulation and reality show FoAM achieves up to 41% higher success rates than strong baselines, with additional insights from VLM-assisted joint inference and robustness analyses. The work provides an open, scalable FoAM benchmark and demonstrates practical gains in generalization to unseen tasks and reliability under real-world disturbances.

Abstract

Multi-task imitation learning (MTIL) has shown significant potential in robotic manipulation by enabling agents to perform various tasks using a single policy. This simplifies the policy deployment and enhances the agent's adaptability across different scenarios. However, key challenges remain, such as maintaining action reliability (e.g., avoiding abnormal action sequences that deviate from nominal task trajectories) and generalizing to unseen tasks with a few expert demonstrations. To address these challenges, we introduce the Foresight-Augmented Manipulation Policy (FoAM), a novel MTIL policy that pioneers the use of multi-modal goal condition as input and introduces a foresight augmentation in addition to the general action reconstruction. FoAM enables the agent to reason about the visual consequences (states) of its actions and learn more expressive embedding that captures nuanced task variations. Extensive experiments on over 100 tasks in simulation and real-world settings demonstrate that FoAM significantly enhances MTIL policy performance, outperforming state-of-the-art baselines by up to 41% in success rate. Meanwhile, we released our simulation suites, including a total of 10 scenarios and over 80 challenging tasks designed for manipulation policy training and evaluation. See the project homepage projFoAM.github.io for project details.

FoAM: Foresight-Augmented Multi-Task Imitation Policy for Robotic Manipulation

TL;DR

FoAM addresses reliability and generalization gaps in multi-task imitation learning by introducing a multi-modal goal-conditioned policy and foresight augmentation, enabling the agent to reason about the consequences of its actions. It fuses language instructions with generated goal images from a fine-tuned Vision-Language Model and learns via a CVAE-transformer policy that predicts action chunks and foresight sequences, optimized with action and foresight losses plus KL regularization. Empirical results across over 100 tasks in simulation and reality show FoAM achieves up to 41% higher success rates than strong baselines, with additional insights from VLM-assisted joint inference and robustness analyses. The work provides an open, scalable FoAM benchmark and demonstrates practical gains in generalization to unseen tasks and reliability under real-world disturbances.

Abstract

Multi-task imitation learning (MTIL) has shown significant potential in robotic manipulation by enabling agents to perform various tasks using a single policy. This simplifies the policy deployment and enhances the agent's adaptability across different scenarios. However, key challenges remain, such as maintaining action reliability (e.g., avoiding abnormal action sequences that deviate from nominal task trajectories) and generalizing to unseen tasks with a few expert demonstrations. To address these challenges, we introduce the Foresight-Augmented Manipulation Policy (FoAM), a novel MTIL policy that pioneers the use of multi-modal goal condition as input and introduces a foresight augmentation in addition to the general action reconstruction. FoAM enables the agent to reason about the visual consequences (states) of its actions and learn more expressive embedding that captures nuanced task variations. Extensive experiments on over 100 tasks in simulation and real-world settings demonstrate that FoAM significantly enhances MTIL policy performance, outperforming state-of-the-art baselines by up to 41% in success rate. Meanwhile, we released our simulation suites, including a total of 10 scenarios and over 80 challenging tasks designed for manipulation policy training and evaluation. See the project homepage projFoAM.github.io for project details.
Paper Structure (13 sections, 4 figures, 3 tables, 2 algorithms)

This paper contains 13 sections, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. METHOD
  4. Pipeline Overview
  5. Fine-tuned Goal Imagination Module
  6. Foresight Augmentation
  7. FoAM Policy Implementation
  8. Experiments
  9. Data Collection
  10. Experiment Results
  11. VLM-FoAM Joint Inference
  12. Robustness Analysis
  13. Conclusion and Future Work

Figures (4)

  • Figure 1: Training and inference pipelines of FoAM. The input terms remain consistent throughout both training and inference. During training, actions and their corresponding consequences are predicted, with the predicted consequences and actions being aligned with the input goal image and expert actions, respectively, to update the parameters of FoAM. During inference, the trained policy is used solely to predict the action $\hat{a}_{t}$.
  • Figure 2: Inference demonstrations of the fine-tuned goal imagination module. The leftmost image illustrates the initial observation, while the next four images represent the edited goal images generated based on the initial observation and the task instruction provided at the top. Please visit the https://projFoAM.github.io/ for more examples.
  • Figure 3: Snapshots of each scenario in the FoAM benchmark. The middle snapshot provides an overview of the simulated dual-arm system we developed in MuJoCo todorov2012mujoco. The tasks in the benchmark are divided into five categories for evaluation: pink for dual-arm tasks, yellow for cabinet-based tasks, green for color-block-based tasks, orange for locker-based tasks, and gray for other tasks. The objects in these scenarios are sourced from dasari2023pgdmXiang_2020_SAPIENMo_2019_CVPRchang2015shapenet. The FoAM benchmark offers high-degree-of-freedom simulation suites. Tutorials for creating custom environments are available on the https://projFoAM.github.io/.
  • Figure 4: Snapshots of the real-world multi-task environment are captured from static externally mounted Orbbec Femto Bolt camera. The tasks include Pick test tubes from the rack, Put fruits into the green bowl, Insert the test tube into holes (four subtasks each), and Place the bitter melon on locker layers (two subtasks). Please refer to the https://projFoAM.github.io/ for videos and more details.