Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In-context Exploration-Exploitation for Reinforcement Learning

Zhenwen Dai, Federico Tomasi, Sina Ghiassian

TL;DR

Through experiments in grid world environments, it is demonstrated that ICEE can learn to solve new RL tasks using only tens of episodes, marking a substantial improvement over the hundreds of episodes needed by the previous in-context learning method.

Abstract

In-context learning is a promising approach for online policy learning of offline reinforcement learning (RL) methods, which can be achieved at inference time without gradient optimization. However, this method is hindered by significant computational costs resulting from the gathering of large training trajectory sets and the need to train large Transformer models. We address this challenge by introducing an In-context Exploration-Exploitation (ICEE) algorithm, designed to optimize the efficiency of in-context policy learning. Unlike existing models, ICEE performs an exploration-exploitation trade-off at inference time within a Transformer model, without the need for explicit Bayesian inference. Consequently, ICEE can solve Bayesian optimization problems as efficiently as Gaussian process biased methods do, but in significantly less time. Through experiments in grid world environments, we demonstrate that ICEE can learn to solve new RL tasks using only tens of episodes, marking a substantial improvement over the hundreds of episodes needed by the previous in-context learning method.

In-context Exploration-Exploitation for Reinforcement Learning

TL;DR

Through experiments in grid world environments, it is demonstrated that ICEE can learn to solve new RL tasks using only tens of episodes, marking a substantial improvement over the hundreds of episodes needed by the previous in-context learning method.

Abstract

In-context learning is a promising approach for online policy learning of offline reinforcement learning (RL) methods, which can be achieved at inference time without gradient optimization. However, this method is hindered by significant computational costs resulting from the gathering of large training trajectory sets and the need to train large Transformer models. We address this challenge by introducing an In-context Exploration-Exploitation (ICEE) algorithm, designed to optimize the efficiency of in-context policy learning. Unlike existing models, ICEE performs an exploration-exploitation trade-off at inference time within a Transformer model, without the need for explicit Bayesian inference. Consequently, ICEE can solve Bayesian optimization problems as efficiently as Gaussian process biased methods do, but in significantly less time. Through experiments in grid world environments, we demonstrate that ICEE can learn to solve new RL tasks using only tens of episodes, marking a substantial improvement over the hundreds of episodes needed by the previous in-context learning method.
Paper Structure (15 sections, 15 equations, 3 figures, 1 algorithm)

This paper contains 15 sections, 15 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Epistemic Uncertainty in Sequence Model Prediction
  4. In-context Policy Learning
  5. Design of return-to-go
  6. Bayesian Optimization Experiments
  7. RL Experiments
  8. Baseline Methods
  9. Evaluation & Results
  10. Conclusion
  11. Derivation of the Objective of a Sequence Model
  12. Derivation of the Unbiased Objective
  13. Implementation Details
  14. Bayesian Optimization Experiments
  15. Quantitative results of the discrete RL experiments

Figures (3)

  • Figure 1: Discrete BO results on 2D functions with 1024 candidate locations. 16 benchmark functions are used with five trials each. The y-axis shows the average distance between the current best estimate and the true minimum. The x-axis of (a) is the number of function evaluations and the x-axis of (b) is the elapsed time.
  • Figure 2: Experimental results of in-context policy learning on grid world RL problems. An agent is expected to solve a game by interacting with the environment for $K$ episodes without online model updates. The y-axis of (a-d) shows the average returns over 100 sampled games after each episode. The y-axis of (e) shows the entropy of the action distribution. The x-axis shows the number of episodes that an agent experiences.
  • Figure 3: Discrete BO results on 2D functions with 1024 candidate locations. The shading area shows the 95% confidence interval of the mean.