PoSSUM: A Protocol for Surveying Social-media Users with Multimodal LLMs

TL;DR

PoSSUM presents an end-to-end protocol to unobtrusively poll social-media users through multimodal LLMs by constructing silicon samples from real-time digital traces. It couples modular prompting and filters with Multilevel Regression and Post-Stratification (MrP) using structured priors to correct for platform-driven Selection Bias and produce population- and sub-population estimates. Validated in the 2024 US Presidential election, PoSSUM achieved state-level predictive accuracy and demonstrated novel learning while revealing limitations in third-party coverage and time-sensitivity due to benchmark noise. The approach offers a fully automated, scalable alternative to traditional surveys, contingent on careful bias mitigation and cross-platform data integration to ensure robust, timely public-opinion insights.

Abstract

This paper introduces PoSSUM, an open-source protocol for unobtrusive polling of social-media users via multimodal Large Language Models (LLMs). PoSSUM leverages users' real-time posts, images, and other digital traces to create silicon samples that capture information not present in the LLM's training data. To obtain representative estimates, PoSSUM employs Multilevel Regression and Post-Stratification (MrP) with structured priors to counteract the observable selection biases of social-media platforms. The protocol is validated during the 2024 U.S. Presidential Election, for which five PoSSUM polls were conducted and published on GitHub and X. In the final poll, fielded October 17-26 with a synthetic sample of 1,054 X users, PoSSUM accurately predicted the outcomes in 50 of 51 states and assigned the Republican candidate a win probability of 0.65. Notably, it also exhibited lower state-level bias than most established pollsters. These results demonstrate PoSSUM's potential as a fully automated, unobtrusive alternative to traditional survey methods.

Figures (32)

• Figure 1: A conceptual description of silicon sampling.
• Figure 2: An overview of the PoSSUM protocol.
• Figure 3: Toy example showing the composition of a prompt under the PoSSUM framework. Red arrows pointing to '[...]' indicate instances where modular prompt components are slotted in. The dotted arrow indicates the LLM generation conditional on a given prompt. The above toy example showcases a single feature ($2020$ vote choice), though multiple features can generally be extracted simultaneously. Not every prompt contains all of the elements indicated in this Figure.
• Figure 4: State-level predictive power on the Republican - Democrat margin. Training data includes highly speculative records. Model fit to the final PoSSUM poll, fielded from the $17^{th}$ to the $26^{th}$ of October.
• Figure 5: State-level predictive power on vote share by candidate. Training data includes highly speculative records. Model fit to pooled dataset of $5$ polls, fielded from the $15^{th}$ of August to the $26^{th}$ of October.