Modeling Object Attention in Mobile AR for Intrinsic Cognitive Security

TL;DR

This work models attention in mobile AR using object recall as a proxy for cognitive focus, addressing potential attention-based security attacks. It pools data from four AR studies (n=1,152 recall probes) and evaluates a PLS-SEM model with formative Object, Scene, and User State composites predicting Object Recall, benchmarked against Random Forests and an MLP. The pooled results show $R^2=0.254$ with the strongest path from User State to recall, and provide per-object recall probabilities that can guide interface mitigations when recall is predicted to fall. The findings offer actionable design levers for AR interfaces—such as highlighting mission-critical objects and simplifying tasks under high load—and demonstrate that PLS-SEM delivers interpretable, predictive insights suitable for intrinsic cognitive security in mobile AR.

Abstract

We study attention in mobile Augmented Reality (AR) using object recall as a proxy outcome. We observe that the ability to recall an object (physical or virtual) that was encountered in a mobile AR experience depends on many possible impact factors and attributes, with some objects being readily recalled while others are not, and some people recalling objects overall much better or worse than others. This opens up a potential cognitive attack in which adversaries might create conditions that make an AR user not recall certain potentially mission-critical objects. We explore whether a calibrated predictor of object recall can help shield against such cognitive attacks. We pool data from four mobile AR studies (with a total of 1,152 object recall probes) and fit a Partial Least Squares Structural Equation Model (PLS-SEM) with formative Object, Scene, and User State composites predicting recall, also benchmarking against Random Forest and multilayer perceptron classifiers. PLS-SEM attains the best F1 score in three of four studies. Additionally, path estimates identify lighting, augmentation density, AR registration stability, cognitive load, and AR familiarity as primary drivers. The model outputs per-object recall probabilities that can drive interface adjustments when predicted recall falls. Overall, PLS-SEM provides competitive accuracy with interpretable levers for design and evaluation in mobile AR.

Figures (2)

• Figure 1: Our overall theorized model. Bold variables are already present in our model. Future data collection will include all listed variables.
• Figure 2: The resulting PLS-SEM model from combining studies 1 to 4. Rectangles denote observed indicators, hexagons denote latent constructs. Arrows from indicators to constructs represent measurement paths with outer weights $w$, where larger $w$ means the indicator contributes more strongly to the construct. Arrows between constructs represent structural paths with coefficients $\beta$; dashed arrows indicate negative paths. $\lambda$ marks the loading of the single-indicator reflective construct. $r^2$ shows the proportion of variance explained in the dependent construct.