VQEL: Enabling Self-Developed Symbolic Language in Agents through Vector Quantization in Emergent Language Games

TL;DR

VQEL, a novel method that incorporates Vector Quantization into the agents' architecture, enabling them to autonomously invent and develop discrete symbolic representations in a self-play referential game, and benefits from improved control and reduced susceptibility to collapse, thanks to the incorporation of vector quantization.

Abstract

In the field of emergent language, efforts have traditionally focused on developing communication protocols through interactions between agents in referential games. However, the aspect of internal language learning, where language serves not only as a communicative tool with others but also as a means for individual thinking, self-reflection, and problem-solving remains underexplored. Developing a language through self-play, without another agent's involvement, poses a unique challenge. It requires an agent to craft symbolic representations and train them using direct gradient methods. The challenge here is that if an agent attempts to learn symbolic representations through self-play using conventional modeling and techniques such as REINFORCE, the solution will offer no advantage over previous multi-agent approaches. We introduce VQEL, a novel method that incorporates Vector Quantization into the agents' architecture, enabling them to autonomously invent and develop discrete symbolic representations in a self-play referential game. Following the self-play phase, agents can enhance their language through reinforcement learning and interactions with other agents in the mutual-play phase. Our experiments across various datasets demonstrate that VQEL not only outperforms the traditional REINFORCE method but also benefits from improved control and reduced susceptibility to collapse, thanks to the incorporation of vector quantization.

Figures (6)

• Figure 1: An overview of agent architecture during self-play. In this phase, the agent uses the Object Encoder and Text Generation modules to produce a sequence of representations from the target image, which are discretized by the VQ module. The agent then employs the Text Perception module to derive a unified representation from this sequence of vectors. Subsequently, it engages in a contrastive loss game on a batch, using the representations obtained from the Object Encoder and Text Perception modules for the images.
• Figure 2: An overview of agents architecture during mutual-play. After completing self-play, the agent engages in mutual play with another agent. In this phase, the sender generates symbols for the input image based on the VQ module and sends them to the receiver. The receiver embeds both the sequence of symbols and the objects and then it is trained using the contrastive loss. The sender also fine-tunes itself with the help of the REINFORCE algorithm, leveraging the receiver's loss.
• Figure 3: Comparison of unique messages generated by VQEL and REINFORCE across datasets. VQEL produces more unique messages with larger vocabularies but sees a reduction post-mutual play; the gap narrows with smaller vocabularies, yet VQEL still excels in accuracy
• Figure 4: Impact of the commitment loss weight on VQEL model performance using the CelebA dataset
• Figure 5: Impact of the length of the message on VQEL model performance using the CelebA dataset