Investigating the Development of Task-Oriented Communication in Vision-Language Models

TL;DR

This work probes whether vision-language models can develop task-oriented communication that departs from natural language. Using zero-shot prompting in referential games, it demonstrates that VLMs can invent efficient, compact languages and even covert protocols that are difficult for external observers to interpret, while remaining usable for model-based coordination. The study shows robust performance gains under constrained descriptions and reveals rich, model-specific linguistic structures through comprehensive analysis across datasets. These findings underscore both the potential to improve multimodal collaboration and the risks related to transparency and alignment in emergent, task-tuned communication systems. The work establishes referential games as a powerful testbed for exploring communication protocols that emerge from pretrained multimodal agents.

Abstract

We investigate whether \emph{LLM-based agents} can develop task-oriented communication protocols that differ from standard natural language in collaborative reasoning tasks. Our focus is on two core properties such task-oriented protocols may exhibit: Efficiency -- conveying task-relevant information more concisely than natural language, and Covertness -- becoming difficult for external observers to interpret, raising concerns about transparency and control. To investigate these aspects, we use a referential-game framework in which vision-language model (VLM) agents communicate, providing a controlled, measurable setting for evaluating language variants. Experiments show that VLMs can develop effective, task-adapted communication patterns. At the same time, they can develop covert protocols that are difficult for humans and external agents to interpret. We also observe spontaneous coordination between similar models without explicitly shared protocols. These findings highlight both the potential and the risks of task-oriented communication, and position referential games as a valuable testbed for future work in this area.

Figures (7)

• Figure 1: Illustration of a referential game using the flags dataset. For clarity, the shared visual world, comprising 4 flag images, is shown on both sides. In step (1), the sender is prompted to develop a lexicon for describing the images. In step (2), the receiver independently develops its own lexicon based on the same visual input. Alternatively, in step (2*), the sender shares its lexicon with the receiver. In step (3), one image is randomly selected as the target, and the sender produces a description using its developed lexicon and communicates it to the receiver. Finally, in step (4), the receiver attempts to match the sender’s description to one of the candidate images using its own visual and language representation.
• Figure 2: Examples from three datasets (Flags, COCO, CLEVR), each showing two images with three descriptions: Natural (top row), Efficient (middle row), and Covert (bottom row).
• Figure 3: Comparing Natural and Efficient Languages generated by GPT across varied description lengths. X-axis: Average number of characters used for target descriptions. Y-axis: Game accuracy. Marker size: Average rate of new words used by the model when describing the targets((range: $0.0$ – $0.97$). Error bars: Standard error of the mean (SEM).
• Figure 4: UMAP projection of natural and task-oriented language variants produced by GPT, Qwen, and Pixtral for the Flags dataset.
• Figure 5: Histogram of human evaluation results. Cyan bars represent the Natural language condition, where participants performed best. Orange bars correspond to the Efficient language condition, which exhibited the highest variance. Green bars indicate the Covert language condition, where participants struggled the most.