XferBench: a Data-Driven Benchmark for Emergent Language
Brendon Boldt, David Mortensen
TL;DR
XferBench introduces a data-driven benchmark to quantify the overall quality of emergent languages by measuring transfer learning effectiveness to human languages. The method pretrains a small language model on an emergent language corpus and evaluates cross-entropy across multiple human languages, aggregating results to yield a single score that reflects similarity to human language within a neural framework. The paper provides empirical evidence across human, synthetic, and emergent baselines, demonstrates correlation with a downstream machine translation task, and offers an accessible Python implementation. This benchmark shifts evaluation from hand-crafted metrics to scalable, data-driven assessment, enabling broader comparisons and future extensions as emergent communication research progresses. The approach has practical impact by offering a scalable tool to gauge the practical utility of emergent languages for real NLP tasks while highlighting limitations related to interface scope and compute requirements.
Abstract
In this paper, we introduce a benchmark for evaluating the overall quality of emergent languages using data-driven methods. Specifically, we interpret the notion of the "quality" of an emergent language as its similarity to human language within a deep learning framework. We measure this by using the emergent language as pretraining data for a downstream NLP tasks in human language -- the better the downstream performance, the better the emergent language. We implement this benchmark as an easy-to-use Python package that only requires a text file of utterances from the emergent language to be evaluated. Finally, we empirically test the benchmark's validity using human, synthetic, and emergent language baselines.