Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inductive Biases for Zero-shot Systematic Generalization in Language-informed Reinforcement Learning

Negin Hashemi Dijujin, Seyed Roozbeh Razavi Rohani, Mohammad Mahdi Samiei, Mahdieh Soleymani Baghshah

TL;DR

The paper tackles the problem of achieving sample-efficient and systematic generalization in language-informed reinforcement learning. It introduces ICMO, an Instruction Conditioned MOdular network that grounds language through a Neural Production System encoder augmented with memory feedback and a language entrance mechanism. Extensive BabyAI experiments show ICMO substantially improves zero-shot generalization, with near-zero Generalization Gap and strong test performance, validated by ablations that pinpoint the benefits of memory-informed, language-grounded processing. The work demonstrates that modular, sparsely connected encoders combined with memory-grounded language interfaces can markedly enhance generalization and training stability in RL, offering a practical route toward robust language-guided agents.

Abstract

Sample efficiency and systematic generalization are two long-standing challenges in reinforcement learning. Previous studies have shown that involving natural language along with other observation modalities can improve generalization and sample efficiency due to its compositional and open-ended nature. However, to transfer these properties of language to the decision-making process, it is necessary to establish a proper language grounding mechanism. One approach to this problem is applying inductive biases to extract fine-grained and informative representations from the observations, which makes them more connectable to the language units. We provide architecture-level inductive biases for modularity and sparsity mainly based on Neural Production Systems (NPS). Alongside NPS, we assign a central role to memory in our architecture. It can be seen as a high-level information aggregator which feeds policy/value heads with comprehensive information and simultaneously guides selective attention in NPS through attentional feedback. Our results in the BabyAI environment suggest that the proposed model's systematic generalization and sample efficiency are improved significantly compared to previous models. An extensive ablation study on variants of the proposed method is conducted, and the effectiveness of each employed technique on generalization, sample efficiency, and training stability is specified.

Inductive Biases for Zero-shot Systematic Generalization in Language-informed Reinforcement Learning

TL;DR

The paper tackles the problem of achieving sample-efficient and systematic generalization in language-informed reinforcement learning. It introduces ICMO, an Instruction Conditioned MOdular network that grounds language through a Neural Production System encoder augmented with memory feedback and a language entrance mechanism. Extensive BabyAI experiments show ICMO substantially improves zero-shot generalization, with near-zero Generalization Gap and strong test performance, validated by ablations that pinpoint the benefits of memory-informed, language-grounded processing. The work demonstrates that modular, sparsely connected encoders combined with memory-grounded language interfaces can markedly enhance generalization and training stability in RL, offering a practical route toward robust language-guided agents.

Abstract

Sample efficiency and systematic generalization are two long-standing challenges in reinforcement learning. Previous studies have shown that involving natural language along with other observation modalities can improve generalization and sample efficiency due to its compositional and open-ended nature. However, to transfer these properties of language to the decision-making process, it is necessary to establish a proper language grounding mechanism. One approach to this problem is applying inductive biases to extract fine-grained and informative representations from the observations, which makes them more connectable to the language units. We provide architecture-level inductive biases for modularity and sparsity mainly based on Neural Production Systems (NPS). Alongside NPS, we assign a central role to memory in our architecture. It can be seen as a high-level information aggregator which feeds policy/value heads with comprehensive information and simultaneously guides selective attention in NPS through attentional feedback. Our results in the BabyAI environment suggest that the proposed model's systematic generalization and sample efficiency are improved significantly compared to previous models. An extensive ablation study on variants of the proposed method is conducted, and the effectiveness of each employed technique on generalization, sample efficiency, and training stability is specified.
Paper Structure (20 sections, 4 equations, 5 figures, 4 tables)

This paper contains 20 sections, 4 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Proposed Model
  4. Problem Formulation
  5. Architecture
  6. Experiments
  7. Setup
  8. The Benchmark for Language-informed Systematic Generalization
  9. Evaluation Metrics
  10. Baselines
  11. Ablations
  12. Results
  13. Discussion
  14. Related Work
  15. Language-informed Studies
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Overall architecture of ICMO (The switch icon () performs index selection; for example, the output of rule selector module is an index $r$ to choose the most relevant rule action, $MLP_r$, and the left port of this switch receives the array and the right port outputs the selected item)
  • Figure 2: Test and train MR trends comparing ICMO to baselines in terms of test MR (a-f) and train MR (g-l)
  • Figure 3: Test and train MR trends comparing instruction ablations
  • Figure 4: Test and train MR trends comparing memory ablations
  • Figure 5: Test-time Radar Charts indicating the overall performance of (a) baseline models and (b) ablation models against ICMO at a glance