Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frog Soup: Zero-Shot, In-Context, and Sample-Efficient Frogger Agents

Xiang Li, Yiyang Hao, Doug Fulop

TL;DR

This study probes zero-shot and in-context capabilities of reasoning LLMs to play the Atari Frogger game and to bootstrap traditional RL. It leverages object-centric representations (OCAtari) and evaluations across multiple models (GPT-4o, Claude-3.7 Sonnet, o3-mini, QwQ-32B) to compare zero-shot, in-context, and LLM-guided RL. Key findings show that careful prompting—particularly 0 past steps with moderate reasoning and context-reward signals—yields strong zero-shot performance, while LLM-generated demonstrations substantially improve sample efficiency for DQN. The work points to future directions in fine-tuning reasoning LLMs and integrating LLM-guided search (MCTS) to further close the gap with game-level planning in RL.

Abstract

One of the primary aspirations in reinforcement learning research is developing general-purpose agents capable of rapidly adapting to and mastering novel tasks. While RL gaming agents have mastered many Atari games, they remain slow and costly to train for each game. In this work, we demonstrate that latest reasoning LLMs with out-of-domain RL post-training can play a challenging Atari game called Frogger under a zero-shot setting. We then investigate the effect of in-context learning and the amount of reasoning effort on LLM performance. Lastly, we demonstrate a way to bootstrap traditional RL method with LLM demonstrations, which significantly improves their performance and sample efficiency. Our implementation is open sourced at https://github.com/AlienKevin/frogger.

Frog Soup: Zero-Shot, In-Context, and Sample-Efficient Frogger Agents

TL;DR

This study probes zero-shot and in-context capabilities of reasoning LLMs to play the Atari Frogger game and to bootstrap traditional RL. It leverages object-centric representations (OCAtari) and evaluations across multiple models (GPT-4o, Claude-3.7 Sonnet, o3-mini, QwQ-32B) to compare zero-shot, in-context, and LLM-guided RL. Key findings show that careful prompting—particularly 0 past steps with moderate reasoning and context-reward signals—yields strong zero-shot performance, while LLM-generated demonstrations substantially improve sample efficiency for DQN. The work points to future directions in fine-tuning reasoning LLMs and integrating LLM-guided search (MCTS) to further close the gap with game-level planning in RL.

Abstract

One of the primary aspirations in reinforcement learning research is developing general-purpose agents capable of rapidly adapting to and mastering novel tasks. While RL gaming agents have mastered many Atari games, they remain slow and costly to train for each game. In this work, we demonstrate that latest reasoning LLMs with out-of-domain RL post-training can play a challenging Atari game called Frogger under a zero-shot setting. We then investigate the effect of in-context learning and the amount of reasoning effort on LLM performance. Lastly, we demonstrate a way to bootstrap traditional RL method with LLM demonstrations, which significantly improves their performance and sample efficiency. Our implementation is open sourced at https://github.com/AlienKevin/frogger.
Paper Structure (31 sections, 1 equation, 11 figures)

This paper contains 31 sections, 1 equation, 11 figures.

Table of Contents

  1. Introduction
  2. Background
  3. RL for Atari Games
  4. Atari Game Selection
  5. Atari 2600 Frogger Game Mechanics
  6. LLM Agents
  7. Spatial Reasoning of Multimodal LLM
  8. Object-Centric Atari
  9. Zero-Shot LLM Agents
  10. Reasoning Effort and Performance
  11. Zero-Shot Performance of Open-source LLM Agent
  12. In-Context LLM Agents
  13. Explorative LLM Agent
  14. Reflective LLM Agent
  15. Depth-First Search Agent
  16. ...and 16 more sections

Figures (11)

  • Figure 1: States, actions, and rewards of the Atari 2600 Frogger game environment
  • Figure 2: Zero-Shot/In-Context RL with o3-mini on Object-Centric Representation of Frogger
  • Figure 3: Episodic rewards vs number of completion tokens with o3-mini and QwQ-32B
  • Figure 4: Enhance DQN with LLM-generated demonstrations
  • Figure 5: LLM-Guided DQN
  • ...and 6 more figures