Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning

Jonathan Cook, Chris Lu, Edward Hughes, Joel Z. Leibo, Jakob Foerster

TL;DR

This work is the first to present general models that achieve emergent cultural accumulation in reinforcement learning, opening up new avenues towards more open-ended learning systems, as well as presenting new opportunities for modelling human culture.

Abstract

Cultural accumulation drives the open-ended and diverse progress in capabilities spanning human history. It builds an expanding body of knowledge and skills by combining individual exploration with inter-generational information transmission. Despite its widespread success among humans, the capacity for artificial learning agents to accumulate culture remains under-explored. In particular, approaches to reinforcement learning typically strive for improvements over only a single lifetime. Generational algorithms that do exist fail to capture the open-ended, emergent nature of cultural accumulation, which allows individuals to trade-off innovation and imitation. Building on the previously demonstrated ability for reinforcement learning agents to perform social learning, we find that training setups which balance this with independent learning give rise to cultural accumulation. These accumulating agents outperform those trained for a single lifetime with the same cumulative experience. We explore this accumulation by constructing two models under two distinct notions of a generation: episodic generations, in which accumulation occurs via in-context learning and train-time generations, in which accumulation occurs via in-weights learning. In-context and in-weights cultural accumulation can be interpreted as analogous to knowledge and skill accumulation, respectively. To the best of our knowledge, this work is the first to present general models that achieve emergent cultural accumulation in reinforcement learning, opening up new avenues towards more open-ended learning systems, as well as presenting new opportunities for modelling human culture.

Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning

TL;DR

This work is the first to present general models that achieve emergent cultural accumulation in reinforcement learning, opening up new avenues towards more open-ended learning systems, as well as presenting new opportunities for modelling human culture.

Abstract

Cultural accumulation drives the open-ended and diverse progress in capabilities spanning human history. It builds an expanding body of knowledge and skills by combining individual exploration with inter-generational information transmission. Despite its widespread success among humans, the capacity for artificial learning agents to accumulate culture remains under-explored. In particular, approaches to reinforcement learning typically strive for improvements over only a single lifetime. Generational algorithms that do exist fail to capture the open-ended, emergent nature of cultural accumulation, which allows individuals to trade-off innovation and imitation. Building on the previously demonstrated ability for reinforcement learning agents to perform social learning, we find that training setups which balance this with independent learning give rise to cultural accumulation. These accumulating agents outperform those trained for a single lifetime with the same cumulative experience. We explore this accumulation by constructing two models under two distinct notions of a generation: episodic generations, in which accumulation occurs via in-context learning and train-time generations, in which accumulation occurs via in-weights learning. In-context and in-weights cultural accumulation can be interpreted as analogous to knowledge and skill accumulation, respectively. To the best of our knowledge, this work is the first to present general models that achieve emergent cultural accumulation in reinforcement learning, opening up new avenues towards more open-ended learning systems, as well as presenting new opportunities for modelling human culture.
Paper Structure (32 sections, 1 equation, 7 figures, 1 table, 3 algorithms)

This paper contains 32 sections, 1 equation, 7 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Partially-Observable Stochastic Games
  4. Partially-Observable Markov Decision Processes
  5. Meta-RL
  6. Generational Training
  7. Problem Statement
  8. Environments
  9. Cultural Accumulation in RL
  10. In-Context Accumulation
  11. Training
  12. Evaluation
  13. In-Weights Accumulation
  14. Results
  15. In-Context Results
  16. ...and 17 more sections

Figures (7)

  • Figure 1: Left: A flow chart describing our RL model of cultural accumulation. Right: An annotated, illustrative plot demonstrating in-context accumulation as observed in our results.
  • Figure 2: Left: A visualization of the Goal Sequence Environment. Right: Routes travelled get shorter across generations in the TSP environment. Visualisation implementation is based on Jumanji bonnet2024jumanji.
  • Figure 3: Left: In-context accumulation during evaluation on Memory Sequence. Right: Evaluation results following training with different oracle accuracies.
  • Figure 4: Left: In-context accumulation during evaluation on Goal Sequence. Right: In-context accumulation during evaluation on TSP.
  • Figure 5: Left: In-weights accumulation on Memory Sequence. Right: In-weights accumulation compounds with resetting. Error bars represent 95% confidence intervals.
  • ...and 2 more figures