Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

InnateCoder: Learning Programmatic Options with Foundation Models

Rubens O. Moraes, Quazi Asif Sadmine, Hendrik Baier, Levi H. S. Lelis

TL;DR

InnateCoder addresses the sample-inefficiency of deep reinforcement learning by using foundation models to extract programmatic options before environment interaction. It decomposes model-generated policies into sub-programs (options) and induces a semantic space over a domain-specific language by replacing sub-trees with options, enabling a mixed syntax-semantic search via stochastic hill climbing. Empirically, InnateCoder achieves superior sample efficiency on MicroRTS and Karel and can scale to thousands of options, outperforming baselines that learn options from environment or rely solely on the foundation model's direct policies. The approach is lightweight and broadly applicable, highlighting foundation models as a practical pre-processing step to bootstrap planning with large, diverse option libraries, and it achieves competitive or superior results against state-of-the-art competition policies.

Abstract

Outside of transfer learning settings, reinforcement learning agents start their learning process from a clean slate. As a result, such agents have to go through a slow process to learn even the most obvious skills required to solve a problem. In this paper, we present InnateCoder, a system that leverages human knowledge encoded in foundation models to provide programmatic policies that encode "innate skills" in the form of temporally extended actions, or options. In contrast to existing approaches to learning options, InnateCoder learns them from the general human knowledge encoded in foundation models in a zero-shot setting, and not from the knowledge the agent gains by interacting with the environment. Then, InnateCoder searches for a programmatic policy by combining the programs encoding these options into larger and more complex programs. We hypothesized that InnateCoder's way of learning and using options could improve the sampling efficiency of current methods for learning programmatic policies. Empirical results in MicroRTS and Karel the Robot support our hypothesis, since they show that InnateCoder is more sample efficient than versions of the system that do not use options or learn them from experience.

InnateCoder: Learning Programmatic Options with Foundation Models

TL;DR

InnateCoder addresses the sample-inefficiency of deep reinforcement learning by using foundation models to extract programmatic options before environment interaction. It decomposes model-generated policies into sub-programs (options) and induces a semantic space over a domain-specific language by replacing sub-trees with options, enabling a mixed syntax-semantic search via stochastic hill climbing. Empirically, InnateCoder achieves superior sample efficiency on MicroRTS and Karel and can scale to thousands of options, outperforming baselines that learn options from environment or rely solely on the foundation model's direct policies. The approach is lightweight and broadly applicable, highlighting foundation models as a practical pre-processing step to bootstrap planning with large, diverse option libraries, and it achieves competitive or superior results against state-of-the-art competition policies.

Abstract

Outside of transfer learning settings, reinforcement learning agents start their learning process from a clean slate. As a result, such agents have to go through a slow process to learn even the most obvious skills required to solve a problem. In this paper, we present InnateCoder, a system that leverages human knowledge encoded in foundation models to provide programmatic policies that encode "innate skills" in the form of temporally extended actions, or options. In contrast to existing approaches to learning options, InnateCoder learns them from the general human knowledge encoded in foundation models in a zero-shot setting, and not from the knowledge the agent gains by interacting with the environment. Then, InnateCoder searches for a programmatic policy by combining the programs encoding these options into larger and more complex programs. We hypothesized that InnateCoder's way of learning and using options could improve the sampling efficiency of current methods for learning programmatic policies. Empirical results in MicroRTS and Karel the Robot support our hypothesis, since they show that InnateCoder is more sample efficient than versions of the system that do not use options or learn them from experience.
Paper Structure (58 sections, 5 equations, 13 figures, 2 tables)

This paper contains 58 sections, 5 equations, 13 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. InnateCoder
  4. Learning Options
  5. Inducing Semantic Spaces with Options
  6. Searching in Semantic Space
  7. Empirical Evaluation
  8. MicroRTS
  9. Karel
  10. Baselines
  11. Foundation Models
  12. Other Specifications
  13. Metrics of Performance
  14. Efficiency Experiment
  15. Competition Experiment
  16. ...and 43 more sections

Figures (13)

  • Figure 1: Top: Context-free grammar specifying a simplified version of the domain-specific language for Karel the Robot. Bottom: Abstract syntax tree for if markersPresent then pickMarker, where MP and PM stand for markersPresent and pickMarker, respectively. Karel is a robot acting on a grid, where it needs to accomplish tasks such as collecting and placing markers on different locations of the grid. In this program, Karel will pick up a marker if one is present in its current location on the grid.
  • Figure 2: InnateCoder has three parts. "Learning Options" harnesses options from a foundation model. The model generates a set of programmatic policies that are broken into a set of options (Section \ref{['sec:options']}). "Semantic Space" uses the options to induce the semantic space of the DSL (Section \ref{['sec:semantic']}). "Local Search" searches in a mixture of the syntax and semantic spaces for a policy $n^*$ (Section \ref{['sec:searching']}).
  • Figure 3: Winning rate (maximum is $100$) per number of games played. The winning rate of the policies each system generates for a given number of games played is computed considering as opponents the policies all systems generate at the end of the learning process. The plots show the average winning rate of 30 independent runs (seeds) and the 95% confidence interval.
  • Figure 4: Average episodic return (maximum is $1.0$ for all tasks) per number of episodes. The plots show average episodic return of 30 independent runs (seeds) and the 95% confidence interval.
  • Figure 5: Average winning rate of InnateCoder policies for different sizes of the option set over 10 independent runs (seeds) of each version. We also present the 95% confidence intervals.
  • ...and 8 more figures

Theorems & Definitions (2)

  • Definition 1: Syntax Space
  • Definition 2: Semantic Space