Learning Next Action Predictors from Human-Computer Interaction

TL;DR

It is argued that learning from the full context of user behavior to anticipate user needs is now a viable task with substantial opportunity.

Abstract

Truly proactive AI systems must anticipate what we will do next. This foresight demands far richer information than the sparse signals we type into our prompts -- it demands reasoning over the entire context of what we see and do. We formalize this as next action prediction (NAP): given a sequence of a user's multimodal interactions with a computer (screenshots, clicks, sensor data), predict that user's next action. Progress on this task requires both new data and modeling approaches. To scale data, we annotate longitudinal, naturalistic computer use with vision-language models. We release an open-source pipeline for performing this labeling on private infrastructure, and label over 360K actions across one month of continuous phone usage from 20 users, amounting to 1,800 hours of screen time. We then introduce LongNAP, a user model that combines parametric and in-context learning to reason over long interaction histories. LongNAP is trained via policy gradient methods to generate user-specific reasoning traces given some context; retrieve relevant traces from a library of past traces; and then apply retrieved traces in-context to predict future actions. Using an LLM-as-judge evaluation metric (0-1 similarity to ground truth), LongNAP significantly outperforms supervised finetuning and prompted baselines on held-out data (by 79% and 39% respectively). Additionally, LongNAP generalizes to held out users when trained across individuals. The space of next actions a user might take at any moment is unbounded, spanning thousands of possible outcomes. Despite this, 17.1% of LongNAP's predicted trajectories are well-aligned with what a user does next (LLM-judge score $\geq$ 0.5). This rises to 26% when we filter to highly confident predictions. In sum, we argue that learning from the full context of user behavior to anticipate user needs is now a viable task with substantial opportunity.

Figures (10)

• Figure 1: Long-Context Next Action Predictors (LongNAPs) draw from the entirety of a user's multimodal context (e.g. screenshots)---retrieving over an unbounded history---to predict what they will do next. We train LongNAP end-to-end on data from 20 users over a month, spanning 1.9M screenshots or 1,800 hours of screen on-time. Predictions are rewarded based on LLM-judged similarity to a set of ground truth future actions.
• Figure 2: LongNAP significantly outperforms all baselines by at least 39.4% relative to the strongest baseline. We evaluate with LLM-judge, which outputs similarity to ground truth future actions (0-1 score). Performance is averaged across 20 models trained on individual users.
• Figure 3: NAPsack enables passive collection of human-computer interaction data. It ingests screenshots and input events, compresses them to retain only meaningful frames, and annotates with action descriptions.
• Figure 4: Predictions from LongNAP are generated in a two-phase process. In the first phase, LongNAP Reasons to Retrieve: conditioned on what the user sees right now (e.g. a set of paper reviews), LongNAP generates a reasoning trace and uses it to retrieve past traces. Using retrieved traces, LongNAP Reasons to Predict: generating a final reasoning trace, adding it back to memory, and then predicting the next steps a user might take. The predicted trajectory is compared against a ground truth (with an LLM-as-a-judge similarity score), and then LongNAP is optimized with GRPO shao2024deepseekmath.
• Figure 5: Some users are more predictable than others. When LongNAP is trained on a single user (in our generalizing over time experiments, \ref{['sec:splits_exps']}), LLM-as-a-judge evals vary substantially from one user's LongNAP to another user's. In the above figure, we re-evaluate across checkpoints from training epochs for each user, highlighting variance.