Just-In-Time Objectives: A General Approach for Specialized AI Interactions

TL;DR

This work introduces Just-In-Time objectives, a framework for inferring in-the-moment user goals from interaction traces and rapidly optimizing downstream AI behavior to meet those goals. It formalizes a generate-and-evaluate loop where a JIT objective guides both content creation and assessment, and instantiates this approach in Poppins, a browser-based UI generation system. Through studies with 14 browser traces and 410 online submissions, the authors show that JIT objectives yield more accurate, useful, and task-aligned outputs than baselines, and that they enable on-demand, personalized tools and interfaces. The work discusses broad implications for human–AI interaction, policy and ethics, and design practices, arguing that explicit, localizable objectives can expand the specificity and relevance of AI assistance while highlighting the tradeoffs in control, privacy, and accountability.

Abstract

Large language models promise a broad set of functions, but when not given a specific objective, they default to milquetoast results such as drafting emails littered with cliches. We demonstrate that inferring the user's in-the-moment objective, then rapidly optimizing for that singular objective, enables LLMs to produce tools, interfaces, and responses that are more responsive and desired. We contribute an architecture for automatically inducing just-in-time objectives by passively observing user behavior, then steering downstream AI systems through generation and evaluation against this objective. Inducing just-in-time objectives (e.g., "Clarify the abstract's research contribution") enables automatic generation of tools, e.g., those that critique a draft based on relevant HCI methodologies, anticipate related researchers' reactions, or surface ambiguous terminology. In a series of experiments (N=14, N=205) on participants' own tasks, JIT objectives enable LLM outputs that achieve 66-86% win rates over typical LLMs, and in-person use sessions (N=17) confirm that JIT objectives produce specialized tools unique to each participant.

Figures (12)

• Figure 1: Applying Just-In-Time Objectives. The core of our architecture is a just-in-time objective that can add on to LLM system components, such as those responsible for generation and evaluation. Here, we show an example of dual generation (gen_objective) and evaluation (eval_objective) steps that optimize model behavior. The generation step produces candidates of the specified output format based on the user context and goal, and the evaluation step assesses these candidates with respect to the goal.
• Figure 2: The JSON specification for just-in-time objectives includes a name, 1-2 sentence description, and weight indicating the estimated importance of the objective (1-10 scale).
• Figure 3: Inducing Just-In-Time Objectives. The core architecture depends on a single upstream step of objective induction that infers user goal(s) based on observed context and instantiates each goal as a just-in-time objective, which can later be applied to optimize generation (gen_objective) and evaluation (eval_objective).
• Figure 4: Poppins incorporates just-in-time objectives to power UI generation. The Poppins system instantiates our just-in-time objectives architecture by inducing objectives from a user's context (e.g., their screen or text document). Then, generation and evaluation steps steer objective application for Tool Generation (Poppins-tools) and Expert Generation (Poppins-experts) to automatically generate a tailored output for the user's task (interactive tools or expert responses, respectively).
• Figure 5: Poppins-experts uses just-in-time objectives to generate relevant model expertise, select an appropriate output format (e.g., Feedback, Brainstorm), and generate model responses drawing on the specified expertise.