CtD: Composition through Decomposition in Emergent Communication

TL;DR

This work tackles compositional generalization in emergent communication by introducing Composition through Decomposition (CtD), a two-step framework that first learns a discrete codebook of concepts via Decompose in a multi-target setting and then composes those concepts to describe unseen images in the Compose step. By constraining the latent space with a codebook and optimizing a dual objective that also enforces codebook alignment, CtD achieves superior compositionality metrics and, in many datasets, perfect accuracy, including zero-shot generalization to novel phrase combinations. The approach is validated across five datasets (Thing, Shape, Mnist, Coco, QRC) and outperforms GS and QT baselines, while demonstrating the critical role of multi-target learning, codebook utilization, and careful multi-loss balancing. The work also provides a nuanced discussion of evaluation metrics for compositionality (AMI, CBM, CI, BOS, POS) and situates CtD within broader literature on discrete bottlenecks, iterated learning, and concept learning for language-like representations. Overall, CtD presents a robust path toward discrete, interpretable, and transferrable compositional language in neural agents with practical implications for scalable communication and grounded language learning.

Abstract

Compositionality is a cognitive mechanism that allows humans to systematically combine known concepts in novel ways. This study demonstrates how artificial neural agents acquire and utilize compositional generalization to describe previously unseen images. Our method, termed "Composition through Decomposition", involves two sequential training steps. In the 'Decompose' step, the agents learn to decompose an image into basic concepts using a codebook acquired during interaction in a multi-target coordination game. Subsequently, in the 'Compose' step, the agents employ this codebook to describe novel images by composing basic concepts into complex phrases. Remarkably, we observe cases where generalization in the `Compose' step is achieved zero-shot, without the need for additional training.

Figures (12)

• Figure 1: CtD training process: Concepts are learned and stored in the codebook during the Decompose step (Left), then composed to describe objects during the Compose step (Right).
• Figure 2: The emergent communicating architecture. Sender network is at the left, Receiver network is at the right, $m$ is the communication channel and $CB_\theta$ is the codebook.
• Figure 3: Accuracy and CBM results for the four compositional datasets (excluding the non-compositional Qrc) and the three communication modes (Gumbel-softmax in green, Quantized in red and Codebook in purple) for the Single-Concept dataset (Left) and the Composite-Phrase dataset (Right).
• Figure 4: Accuracy, AMI, CBM, BOS and CI (Y-axis) versus number of targets (X-axis). Illustrating Decompose step for GS (Left) QT (middle) and CB (right) communication on the Thing game.
• Figure 5: Left: Codebook utilization for the 17 Shape concepts trained on a Composite-Phrase dataset and evaluated with CBM on 1,000 test samples, achieving a perfect score. Right: Heatmap of code-word similarities from the same run.