Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Building Open-Ended Embodied Agent via Language-Policy Bidirectional Adaptation

Shaopeng Zhai, Jie Wang, Tianyi Zhang, Fuxian Huang, Qi Zhang, Ming Zhou, Jing Hou, Yu Qiao, Yu Liu

TL;DR

OpenPAL tackles open-ended embodied agents by bi-directionally aligning a language-model planner with a goal-conditioned RL policy in two stages. Stage I pre-trains the LLM to generate instruction-aligned goals and trains a goal-conditioned policy, while Stage II co-trains them to achieve instruction open-endedness through RL with agent feedback and intrinsic reward shaping. The approach is validated in Contra, showing improved goal completion, generalization to unseen goals, and robust LLM-generated goals, with a detailed LLM fine-tuning and co-training protocol. This framework advances practical human–AI collaboration for open-ended tasks, while acknowledging limitations in goal-space scope and modality expansion for future work.

Abstract

Building embodied agents on integrating Large Language Models (LLMs) and Reinforcement Learning (RL) have revolutionized human-AI interaction: researchers can now leverage language instructions to plan decision-making for open-ended tasks. However, existing research faces challenges in meeting the requirement of open-endedness. They typically either train LLM/RL models to adapt to a fixed counterpart, limiting exploration of novel skills and hindering the efficacy of human-AI interaction. To this end, we present OpenPAL, a co-training framework comprising two stages: (1) fine-tuning a pre-trained LLM to translate human instructions into goals for planning, and goal-conditioned training a policy for decision-making; (2) co-training to align the LLM and policy, achieving instruction open-endedness. We conducted experiments using Contra, an open-ended FPS game, demonstrating that an agent trained with OpenPAL not only comprehends arbitrary instructions but also exhibits efficient execution. These results suggest that OpenPAL holds the potential to construct open-ended embodied agents in practical scenarios.

Building Open-Ended Embodied Agent via Language-Policy Bidirectional Adaptation

TL;DR

OpenPAL tackles open-ended embodied agents by bi-directionally aligning a language-model planner with a goal-conditioned RL policy in two stages. Stage I pre-trains the LLM to generate instruction-aligned goals and trains a goal-conditioned policy, while Stage II co-trains them to achieve instruction open-endedness through RL with agent feedback and intrinsic reward shaping. The approach is validated in Contra, showing improved goal completion, generalization to unseen goals, and robust LLM-generated goals, with a detailed LLM fine-tuning and co-training protocol. This framework advances practical human–AI collaboration for open-ended tasks, while acknowledging limitations in goal-space scope and modality expansion for future work.

Abstract

Building embodied agents on integrating Large Language Models (LLMs) and Reinforcement Learning (RL) have revolutionized human-AI interaction: researchers can now leverage language instructions to plan decision-making for open-ended tasks. However, existing research faces challenges in meeting the requirement of open-endedness. They typically either train LLM/RL models to adapt to a fixed counterpart, limiting exploration of novel skills and hindering the efficacy of human-AI interaction. To this end, we present OpenPAL, a co-training framework comprising two stages: (1) fine-tuning a pre-trained LLM to translate human instructions into goals for planning, and goal-conditioned training a policy for decision-making; (2) co-training to align the LLM and policy, achieving instruction open-endedness. We conducted experiments using Contra, an open-ended FPS game, demonstrating that an agent trained with OpenPAL not only comprehends arbitrary instructions but also exhibits efficient execution. These results suggest that OpenPAL holds the potential to construct open-ended embodied agents in practical scenarios.
Paper Structure (46 sections, 13 equations, 12 figures, 17 tables, 2 algorithms)

This paper contains 46 sections, 13 equations, 12 figures, 17 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Goal-conditioned Reinforcement Learning.
  4. LLMs and Human-AI Interaction.
  5. The Contra: A Battle Royale FPS Game
  6. A Co-training Framework: OpenPAL
  7. Exploring Basic Skills via Non-goal RL
  8. Learning a Goal-conditioned Policy
  9. Open-ended Goal Generation.
  10. Intrinsic Reward Shaping.
  11. Avoiding Policy Degeneration.
  12. Fine-tuning a LLM-based Goal Generator
  13. Dataset Construction.
  14. Multi-step Fine-tuning.
  15. Collaborative Training
  16. ...and 31 more sections

Figures (12)

  • Figure 1: Overview of co-training in OpenPAL. The Policy and LLM is pre-trained with multi-step fine-tuning and goal-conditioned RL, respectively. Then, the co-training aligns them towards achieving instruction open-endedness.
  • Figure 2: (a) The goal completion rate on training dataset; (b) The goal completion rate on unseen goals, i.e., the test dataset; (c) The evaluation of policy learning in cases of w/ and w/o KL-divergence regularizer.
  • Figure 3: (a) The completion ratio of goals with dimension size ranges from 1 to 7; (b) The goal completion ratio of goals that $|g|=3$, the trend curve reflects the improving completion ratio; (c) The sub-goals distribution changes along the training in one loop of co-training, where the description of each $g^i$ is included in \ref{['tab:frequency']}.
  • Figure 4: Preprocessing for an observation with four types of features.
  • Figure 5: Network structure of our proposed policy.
  • ...and 7 more figures