PREFAB: PREFerence-based Affective Modeling for Low-Budget Self-Annotation

TL;DR

PREFAB tackles the high cognitive burden of full-session retrospective self-annotation by focusing on affective inflection regions selected via a peak-end rule-inspired, ordinal preference-learning approach. It employs a Siamese transformer with FiLM conditioning to predict relative arousal changes, detects inflection points, and uses a linear interpolation scheme to reconstruct the full affective trajectory, guided by 5-second inflection windows. In technical evaluation, PREFAB outperforms baselines in identifying inflection regions while maintaining similar overall annotation effort, and in a user study with 25 participants, PREFAB with preview reduces workload and enhances annotator confidence without sacrificing annotation quality. The findings demonstrate a practical, human-centered path to scalable affective data collection, with trade-offs around preview time and opportunities for richer, context-aware extensions.

Abstract

Self-annotation is the gold standard for collecting affective state labels in affective computing. Existing methods typically rely on full annotation, requiring users to continuously label affective states across entire sessions. While this process yields fine-grained data, it is time-consuming, cognitively demanding, and prone to fatigue and errors. To address these issues, we present PREFAB, a low-budget retrospective self-annotation method that targets affective inflection regions rather than full annotation. Grounded in the peak-end rule and ordinal representations of emotion, PREFAB employs a preference-learning model to detect relative affective changes, directing annotators to label only selected segments while interpolating the remainder of the stimulus. We further introduce a preview mechanism that provides brief contextual cues to assist annotation. We evaluate PREFAB through a technical performance study and a 25-participant user study. Results show that PREFAB outperforms baselines in modeling affective inflections while mitigating workload (and conditionally mitigating temporal burden). Importantly PREFAB improves annotator confidence without degrading annotation quality.

Figures (13)

• Figure 1: Visualization of BCE and OCE probability functions. BCE produces a single sigmoid transition between two classes, while OCE partitions the latent score $p_{ij}$ into three ordinal regions using cutpoints $c_0$ and $c_1$.
• Figure 2: Evaluation and results of dynamic time warping (DTW)-based clustering. The left part of the figure shows the trade-off between the cluster silhouette score and data distribution (entropy). The right part of the figure visualizes the four resulting clusters of arousal change patterns ($k=4$): individual time series and mean values are depicted in in gray and red color, respectively.
• Figure 3: Architecture of the PREFAB model. The model takes two consecutive segments sampled at a 1-second interval, each consisting of 12 image frames, game log features, and the player’s biographical information. Both segments are processed by the same encoder in a Siamese network structure. In the inference stage, a single segment is fed into the model to predict the relative value $p$ at each time point.
• Figure 4: Model performance across nine games. The top panels (3×3) report F1 scores (higher values are better) for each method, and the bottom panels (3×3) report corresponding $\Delta$TE (lower values are better).
• Figure 5: Comparison of inflection point sampling across methods. X-axis indicates timestamps and Y-axis arousal levels. Green stars mark inflection points predicted by each method, with green shaded regions denoting their expanded inflection regions. Red stars and regions indicate the corresponding ground truth (GT). For model-based methods (PREFAB and regression), dashed lines show reconstructed trajectories from trained models.