Learning to Communicate Across Modalities: Perceptual Heterogeneity in Multi-Agent Systems

TL;DR

This work investigates how perceptual heterogeneity across modalities influences emergent communication in a heterogeneous multi-step binary referential game. It compares unimodal and multimodal agents, analyzes efficiency, consistency, grounding, and interoperability, and uses bit perturbation and clustering to reveal distributional rather than compositional semantics. Key findings show that perceptual misalignment increases required communication and decoding uncertainty, yet sender messages remain grounded in the sender’s perceptual space; limited fine-tuning enables cross-system communication between differently grounded agents. The results advance our understanding of how representations adapt and transfer across heterogeneous perceptual worlds, with implications for robotics, embodied language grounding, and theories of perceptual embodiment.

Abstract

Emergent communication offers insight into how agents develop shared structured representations, yet most research assumes homogeneous modalities or aligned representational spaces, overlooking the perceptual heterogeneity of real-world settings. We study a heterogeneous multi-step binary communication game where agents differ in modality and lack perceptual grounding. Despite perceptual misalignment, multimodal systems converge to class-consistent messages grounded in perceptual input. Unimodal systems communicate more efficiently, using fewer bits and achieving lower classification entropy, while multimodal agents require greater information exchange and exhibit higher uncertainty. Bit perturbation experiments provide strong evidence that meaning is encoded in a distributional rather than compositional manner, as each bit's contribution depends on its surrounding pattern. Finally, interoperability analyses show that systems trained in different perceptual worlds fail to directly communicate, but limited fine-tuning enables successful cross-system communication. This work positions emergent communication as a framework for studying how agents adapt and transfer representations across heterogeneous modalities, opening new directions for both theory and experimentation.

Figures (10)

• Figure 1: Illustration of the multi-step referential game, where a Sender encodes an audio input into a discrete message and a Receiver uses it to identify the correct visual target among a set of distractors, exchanging messages until a decision is made.
• Figure 2: Cosine similarity matrices of Sender message embeddings for the unimodal and multimodal systems at message length 10. Sender messages within classes remain consistent in both unimodal and multimodal setups.
• Figure 3: Per-class accuracy when flipping variable (fluctuating) bits. Variable bit flips have minimal impact compared to constant bit flips, with a notably larger effect in the unimodal setup.
• Figure 4: Per-class accuracy and variance when flipping an increasing number of constant bits: Flipping 0s to 1s generally leads to a steeper decline in accuracy than the reverse. The similar overall trends between flipping constant 0s and 1s suggest that both carry essential class information. A sharp spike in variance around the third bit flip when perturbing 1s indicates that certain always active bits encode disproportionately critical information.
• Figure 5: t-SNE plots of Class 2 (heart class) messages across varying frequencies and amplitudes. Messages corresponding to lower frequencies exhibit a consistent underlying pattern, while variations in amplitude show minimal impact on message structure.