Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy Improvement using Language Feedback Models

Victor Zhong, Dipendra Misra, Xingdi Yuan, Marc-Alexandre Côté

TL;DR

Language Feedback Models that identify desirable behaviour - actions that help achieve tasks specified in the instruction - for imitation learning in instruction following improve in task-completion rate over strong behavioural cloning baselines on three distinct language grounding environments.

Abstract

We introduce Language Feedback Models (LFMs) that identify desirable behaviour - actions that help achieve tasks specified in the instruction - for imitation learning in instruction following. To train LFMs, we obtain feedback from Large Language Models (LLMs) on visual trajectories verbalized to language descriptions. First, by using LFMs to identify desirable behaviour to imitate, we improve in task-completion rate over strong behavioural cloning baselines on three distinct language grounding environments (Touchdown, ScienceWorld, and ALFWorld). Second, LFMs outperform using LLMs as experts to directly predict actions, when controlling for the number of LLM output tokens. Third, LFMs generalize to unseen environments, improving task-completion rate by 3.5-12.0% through one round of adaptation. Finally, LFM can be modified to provide human-interpretable feedback without performance loss, allowing human verification of desirable behaviour for imitation learning.

Policy Improvement using Language Feedback Models

TL;DR

Language Feedback Models that identify desirable behaviour - actions that help achieve tasks specified in the instruction - for imitation learning in instruction following improve in task-completion rate over strong behavioural cloning baselines on three distinct language grounding environments.

Abstract

We introduce Language Feedback Models (LFMs) that identify desirable behaviour - actions that help achieve tasks specified in the instruction - for imitation learning in instruction following. To train LFMs, we obtain feedback from Large Language Models (LLMs) on visual trajectories verbalized to language descriptions. First, by using LFMs to identify desirable behaviour to imitate, we improve in task-completion rate over strong behavioural cloning baselines on three distinct language grounding environments (Touchdown, ScienceWorld, and ALFWorld). Second, LFMs outperform using LLMs as experts to directly predict actions, when controlling for the number of LLM output tokens. Third, LFMs generalize to unseen environments, improving task-completion rate by 3.5-12.0% through one round of adaptation. Finally, LFM can be modified to provide human-interpretable feedback without performance loss, allowing human verification of desirable behaviour for imitation learning.
Paper Structure (43 sections, 2 equations, 2 figures, 10 tables, 3 algorithms)

This paper contains 43 sections, 2 equations, 2 figures, 10 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Language grounded instruction following.
  4. Online imitation learning.
  5. Behavioural cloning.
  6. Language Feedback Model
  7. Verbalization
  8. Learning a feedback model
  9. Naively learning from LLM feedback.
  10. Efficiently learning a language feedback model.
  11. Learning from language feedback.
  12. Related Work
  13. Instruction following in grounded environments.
  14. LLM agents in language settings.
  15. Learning from feedback.
  16. ...and 28 more sections

Figures (2)

  • Figure 1: Given an environment and instructions to follow, we assume a verbalization procedure that converts observations to language descriptions. Policy improvement using Language Feedback Model involves (a) training a feedback model, then (b) using it to identify desirable behaviour for policy improvement via imitation learning. The feedback model is yellow, other models purple, and generated intermediate data green. An example of LFM-identified behaviour is shown in (c).
  • Figure 2: An example verbalization for Touchdown. We align Clip image embeddings of panorama patches and language embeddings of common noun-phrases to populate a language template. Appendix \ref{['app:touchdown_verbalization']} describes this procedure in detail. The blue arrow at the top indicate the agent's orientation while the green arrows indicate valid directions to proceed in.