Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication

Łukasz Kuciński, Tomasz Korbak, Paweł Kołodziej, Piotr Miłoś

TL;DR

It is proved that compositionality spontaneously arises in the signaling games, where agents communicate over a noisy channel, and a range of noise levels, which depends on the model and the data, indeed promotes compositionality.

Abstract

Communication is compositional if complex signals can be represented as a combination of simpler subparts. In this paper, we theoretically show that inductive biases on both the training framework and the data are needed to develop a compositional communication. Moreover, we prove that compositionality spontaneously arises in the signaling games, where agents communicate over a noisy channel. We experimentally confirm that a range of noise levels, which depends on the model and the data, indeed promotes compositionality. Finally, we provide a comprehensive study of this dependence and report results in terms of recently studied compositionality metrics: topographical similarity, conflict count, and context independence.

Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication

TL;DR

It is proved that compositionality spontaneously arises in the signaling games, where agents communicate over a noisy channel, and a range of noise levels, which depends on the model and the data, indeed promotes compositionality.

Abstract

Communication is compositional if complex signals can be represented as a combination of simpler subparts. In this paper, we theoretically show that inductive biases on both the training framework and the data are needed to develop a compositional communication. Moreover, we prove that compositionality spontaneously arises in the signaling games, where agents communicate over a noisy channel. We experimentally confirm that a range of noise levels, which depends on the model and the data, indeed promotes compositionality. Finally, we provide a comprehensive study of this dependence and report results in terms of recently studied compositionality metrics: topographical similarity, conflict count, and context independence.

Paper Structure

This paper contains 40 sections, 4 theorems, 26 equations, 31 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Noisy channel method
  4. Language and compositionality
  5. Inductive biases and compositionality
  6. Compositionality and communication over a noisy channel
  7. Experiments setup
  8. Differences between experimental and theoretical setups
  9. Training pipeline
  10. Datasets
  11. Training details
  12. Compositionality measures
  13. Experiment results
  14. Main experiment: emergence of compositionaliy
  15. Influence of model inductive bias
  16. ...and 25 more sections

Key Result

Theorem 1

For a uniform distribution, $\mu$ on $\mathcal{F}$ and a permutation $\pi\colon\mathcal{F}\to \mathcal{F}$, the distribution $\mu\circ\pi^{-1}$ is also uniform.

Figures (31)

  • Figure 1: Language, features, and compositionality.
  • Figure 2: Top: shapes3d. Middle: obverter. Bottom: Scrambled shapes3d ($32$, $16$, $8$).
  • Figure 3: Expected value of topographic similarity for a random bijective language with message length $2$, as a function of the alphabet size. "min", "max", and "avg" stand for different ways of computing ranks.
  • Figure 4: Results of the main experiment on shapes3d dataset. Top panel: average value of metrics for various noise levels. The shaded area corresponds to bootstrapped $95\%$-confidence intervals. Bottom panel: kernel density estimators for metrics and noise levels across seeds. Here topo stands for topographic similarity, conf for conflict count, cont for context independence, pos for positional disentanglement, and acc for accuracy.
  • Figure 5: Topographic similarity for experiments in Sections \ref{['sec:model_biases']} and Section \ref{['sec:data_biases']}. The shaded areas correspond to bootstrapped $95\%$-confidence intervals for average topo.
  • ...and 26 more figures

Theorems & Definitions (6)

  • Theorem 1
  • Theorem 2
  • Lemma 1
  • proof
  • Theorem 3
  • proof