Saying the Unsaid: Revealing the Hidden Language of Multimodal Systems Through Telephone Games

TL;DR

The work tackles the opacity of hidden representations in closed-source multimodal systems by introducing a test-time telephone game framework that exploits the systems' preference biases to reveal concept connections. It defines a co-occurrence frequency metric and builds Telescope, a large dataset of concept pairs, to map the hidden language as a scalable world map of concept relationships. Empirical results show that the hidden-language signal is not well captured by semantic or visual similarity, is partially consistent across different systems, and can be reinforced via intermediary concepts to stabilize fragile connections. The approach leverages Reasoning-LLMs as probes to interpret the emergent relationships, offering a practical path toward interpretability and controllability of multimodal systems at test time.

Abstract

Recent closed-source multimodal systems have made great advances, but their hidden language for understanding the world remains opaque because of their black-box architectures. In this paper, we use the systems' preference bias to study their hidden language: During the process of compressing the input images (typically containing multiple concepts) into texts and then reconstructing them into images, the systems' inherent preference bias introduces specific shifts in the outputs, disrupting the original input concept co-occurrence. We employ the multi-round "telephone game" to strategically leverage this bias. By observing the co-occurrence frequencies of concepts in telephone games, we quantitatively investigate the concept connection strength in the understanding of multimodal systems, i.e., "hidden language." We also contribute Telescope, a dataset of 10,000+ concept pairs, as the database of our telephone game framework. Our telephone game is test-time scalable: By iteratively running telephone games, we can construct a global map of concept connections in multimodal systems' understanding. Here we can identify preference bias inherited from training, assess generalization capability advancement, and discover more stable pathways for fragile concept connections. Furthermore, we use Reasoning-LLMs to uncover unexpected concept relationships that transcend textual and visual similarities, inferring how multimodal systems understand and simulate the world. This study offers a new perspective on the hidden language of multimodal systems and lays the foundation for future research on the interpretability and controllability of multimodal systems.

Figures (13)

• Figure 1: Example 5-round telephone games using the latest SOTA multimodal systems released on 2025.3.25. In each image reconstruction, the system prefers stronger nearby concept connections in multimodal systems' understanding, then changing the outputs. (Extended results on this example are provided in Appendix \ref{['sec:appendix/visual']}), and more examples can be found in Appendix \ref{['sec:exp/gen4o']}
• Figure 2: The longevity of concepts combinations in the telephone game (i.e., their co-occurrence frequency) quantitatively reflects the concept connections in multimodal systems' hidden space, termed the "hidden language." (Lighter color means the weaker connection)
• Figure 3: The workflow of telephone game. LLMs convert concept pairs into the start description for telephone game. Then it enters the cycle of text-to-image and image-to-text.
• Figure 4: Visualization of several telephone game examples, reflecting which concepts are connected strongly or weakly in the hidden space of multimodal systems, as well as the intermediary concepts that build stronger connections. For more results, see Appendix \ref{['sec:appendix/visual']}.
• Figure 5: Our scalable framework: (a) Basic Connection: we reveal concept connections and identify nearby keywords; (b) Local Connection: repeated telephone games establish a local graph around a concept; (c) Global Connection: increasing telephone games connect local structures, forming a comprehensive "world map" of the multimodal hidden language!