Situation Graph Prediction: Structured Perspective Inference for User Modeling

TL;DR

This work formalizes Situation Graph Prediction (SGP) as recovering ontology-aligned perspective graphs $G_t$ from multimodal artifacts $X_t$, grounding the task in the DOLCE Ultralite ontology. To enable learning without real user labels, it proposes a structure-first synthetic data generation pipeline that ensures $X_t$ is aligned with $G_t$, enabling controlled, privacy-preserving evaluation. A pilot study using GPT-4o demonstrates that while retrieval-augmented in-context learning improves surface-element grounding, inferring latent states (goals, affect, context) from artifacts remains noticeably harder, revealing a non-trivial latent-surface gap. The results validate SGP as a meaningful and challenging task for structured, transparent perspective inference and highlight the potential of synthetic supervision for privacy-preserving PAi research.

Abstract

Perspective-Aware AI requires modeling evolving internal states--goals, emotions, contexts--not merely preferences. Progress is limited by a data bottleneck: digital footprints are privacy-sensitive and perspective states are rarely labeled. We propose Situation Graph Prediction (SGP), a task that frames perspective modeling as an inverse inference problem: reconstructing structured, ontology-aligned representations of perspective from observable multimodal artifacts. To enable grounding without real labels, we use a structure-first synthetic generation strategy that aligns latent labels and observable traces by design. As a pilot, we construct a dataset and run a diagnostic study using retrieval-augmented in-context learning as a proxy for supervision. In our study with GPT-4o, we observe a gap between surface-level extraction and latent perspective inference--indicating latent-state inference is harder than surface extraction under our controlled setting. Results suggest SGP is non-trivial and provide evidence for the structure-first data synthesis strategy.