Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLF-Bench: Benchmark for Interactive Learning from Language Feedback

Ching-An Cheng, Andrey Kolobov, Dipendra Misra, Allen Nie, Adith Swaminathan

TL;DR

This work introduces LLF-Bench, a benchmark to evaluate how AI agents learn interactively from language feedback rather than from scalar rewards. It presents eight diverse task sets with varied action spaces and horizons, along with a unified OpenAI Gym interface, paraphrase-based instruction and feedback randomization, and multiple feedback types to study robustness and learning efficiency. By treating task instructions as environment-embedded signals and enabling text-mode interactions, LLF-Bench enables evaluation of general, language-grounded learning capabilities in LLM-based agents across multiple domains. The platform also situates itself relative to existing grounded language, text-based games, and LLF benchmarks, arguing that this language-centric, paraphrase-robust framework is essential for developing versatile, human-like learning agents across tasks.

Abstract

We introduce a new benchmark, LLF-Bench (Learning from Language Feedback Benchmark; pronounced as "elf-bench"), to evaluate the ability of AI agents to interactively learn from natural language feedback and instructions. Learning from language feedback (LLF) is essential for people, largely because the rich information this feedback provides can help a learner avoid much of trial and error and thereby speed up the learning process. Large Language Models (LLMs) have recently enabled AI agents to comprehend natural language -- and hence AI agents can potentially benefit from language feedback during learning like humans do. But existing interactive benchmarks do not assess this crucial capability: they either use numeric reward feedback or require no learning at all (only planning or information retrieval). LLF-Bench is designed to fill this omission. LLF-Bench is a diverse collection of sequential decision-making tasks that includes user recommendation, poem writing, navigation, and robot control. The objective of an agent is to interactively solve these tasks based on their natural-language instructions and the feedback received after taking actions. Crucially, to ensure that the agent actually "learns" from the feedback, LLF-Bench implements several randomization techniques (such as paraphrasing and environment randomization) to ensure that the task isn't familiar to the agent and that the agent is robust to various verbalizations. In addition, LLF-Bench provides a unified OpenAI Gym interface for all its tasks and allows the users to easily configure the information the feedback conveys (among suggestion, explanation, and instantaneous performance) to study how agents respond to different types of feedback. Together, these features make LLF-Bench a unique research platform for developing and testing LLF agents.

LLF-Bench: Benchmark for Interactive Learning from Language Feedback

TL;DR

This work introduces LLF-Bench, a benchmark to evaluate how AI agents learn interactively from language feedback rather than from scalar rewards. It presents eight diverse task sets with varied action spaces and horizons, along with a unified OpenAI Gym interface, paraphrase-based instruction and feedback randomization, and multiple feedback types to study robustness and learning efficiency. By treating task instructions as environment-embedded signals and enabling text-mode interactions, LLF-Bench enables evaluation of general, language-grounded learning capabilities in LLM-based agents across multiple domains. The platform also situates itself relative to existing grounded language, text-based games, and LLF benchmarks, arguing that this language-centric, paraphrase-robust framework is essential for developing versatile, human-like learning agents across tasks.

Abstract

We introduce a new benchmark, LLF-Bench (Learning from Language Feedback Benchmark; pronounced as "elf-bench"), to evaluate the ability of AI agents to interactively learn from natural language feedback and instructions. Learning from language feedback (LLF) is essential for people, largely because the rich information this feedback provides can help a learner avoid much of trial and error and thereby speed up the learning process. Large Language Models (LLMs) have recently enabled AI agents to comprehend natural language -- and hence AI agents can potentially benefit from language feedback during learning like humans do. But existing interactive benchmarks do not assess this crucial capability: they either use numeric reward feedback or require no learning at all (only planning or information retrieval). LLF-Bench is designed to fill this omission. LLF-Bench is a diverse collection of sequential decision-making tasks that includes user recommendation, poem writing, navigation, and robot control. The objective of an agent is to interactively solve these tasks based on their natural-language instructions and the feedback received after taking actions. Crucially, to ensure that the agent actually "learns" from the feedback, LLF-Bench implements several randomization techniques (such as paraphrasing and environment randomization) to ensure that the task isn't familiar to the agent and that the agent is robust to various verbalizations. In addition, LLF-Bench provides a unified OpenAI Gym interface for all its tasks and allows the users to easily configure the information the feedback conveys (among suggestion, explanation, and instantaneous performance) to study how agents respond to different types of feedback. Together, these features make LLF-Bench a unique research platform for developing and testing LLF agents.
Paper Structure (25 sections, 3 figures, 2 tables)

This paper contains 25 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Highlights of LLF-Bench
  3. Related Setups and Benchmarks
  4. LLF: Learning from Language Feedback
  5. Setup
  6. Motivation
  7. LLF and RL with text/language observations
  8. Gym Interface of LLF-Bench
  9. Instruction and Feedback
  10. Instruction and Feedback Randomization
  11. Tasks in LLF-Bench
  12. llf-bandit
  13. llf-poem
  14. llf-reco-movie
  15. llf-optimization
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Shows an example navigation task to illustrate our setup, Learning from Language Feedback (LLF). A single episode in LLF starts with a given instruction and can be multi-step long. The actions are taken by the agent that changes the observation and provides a text feedback to the agent. The agent receives no reward or any other form of feedback.
  • Figure 2: LLF-Bench ("Elf-bench") includes 8 sets of LLF problems. Image by Bing Chat.
  • Figure 3: Comparison between RL and LLF setups. LLF replaces reward feedback in RL with language feedback and generalizes the reward maximization objective to general instructions that can be specified via natural language.