FutureVision: A methodology for the investigation of future cognition
Tiago Timponi Torrent, Mark Turner, Nicolás Hinrichs, Frederico Belcavello, Igor Lourenço, Arthur Lorenzi Almeida, Marcelo Viridiano, Ely Edison Matos
TL;DR
FutureVision tackles how people understand multimodal communications about future scenarios by integrating multimodal frame-semantic annotation with eye-tracking. The methodology is demonstrated in a pilot study using fictional ads categorized by valence and counterfactual distance, analyzing both gaze data and semantic representations. Findings suggest that far-future and pessimistic scenarios impose greater cognitive load, evidenced by longer fixations, more saccades, and partial mediation by pupil dilation, with individual differences shaping processing. The work advances understanding of how base-space persistence and fractures influence future cognition and offers a framework for cognitive-state-aware design in future-oriented communication.
Abstract
This paper presents a methodology combining multimodal semantic analysis with an eye-tracking experimental protocol to investigate the cognitive effort involved in understanding the communication of future scenarios. To demonstrate the methodology, we conduct a pilot study examining how visual fixation patterns vary during the evaluation of valence and counterfactuality in fictional ad pieces describing futuristic scenarios, using a portable eye tracker. Participants eye movements are recorded while evaluating the stimuli and describing them to a conversation partner. Gaze patterns are analyzed alongside semantic representations of the stimuli and participants descriptions, constructed from a frame semantic annotation of both linguistic and visual modalities. Preliminary results show that far-future and pessimistic scenarios are associated with longer fixations and more erratic saccades, supporting the hypothesis that fractures in the base spaces underlying the interpretation of future scenarios increase cognitive load for comprehenders.