Mobilizing Waldo: Evaluating Multimodal AI for Public Mobilization

TL;DR

This study assesses the capabilities and risks of multimodal LLMs in public mobilization contexts by testing GPT-4o on a controlled set of crowded scenes drawn from the Hey-Waldo dataset. It employs a carefully designed prompting framework to jointly perform scene description, Waldo localization, and generation of engagement strategies for hypothetical agents, while avoiding real-person privacy concerns. The key finding is that GPT-4o can produce vivid descriptions and creative strategies but struggles with precise spatial localization and accurate social interpretation in high-density scenes, highlighting a gap between language fluency and grounded visual understanding. The work contributes a practical, ethically sound evaluation framework for multimodal AI in social contexts and identifies directions for improving safety and reliability through hybrid architectures and better multimodal grounding.

Abstract

Advancements in multimodal Large Language Models (LLMs), such as OpenAI's GPT-4o, offer significant potential for mediating human interactions across various contexts. However, their use in areas such as persuasion, influence, and recruitment raises ethical and security concerns. To evaluate these models ethically in public influence and persuasion scenarios, we developed a prompting strategy using "Where's Waldo?" images as proxies for complex, crowded gatherings. This approach provides a controlled, replicable environment to assess the model's ability to process intricate visual information, interpret social dynamics, and propose engagement strategies while avoiding privacy concerns. By positioning Waldo as a hypothetical agent tasked with face-to-face mobilization, we analyzed the model's performance in identifying key individuals and formulating mobilization tactics. Our results show that while the model generates vivid descriptions and creative strategies, it cannot accurately identify individuals or reliably assess social dynamics in these scenarios. Nevertheless, this methodology provides a valuable framework for testing and benchmarking the evolving capabilities of multimodal LLMs in social contexts.