JIR-Arena: The First Benchmark Dataset for Just-in-time Information Recommendation

TL;DR

This work formalizes Just-in-time Information Recommendation (JIR) as a decision-making problem and introduces JIR-Arena, the first multimodal benchmark dataset for evaluating JIR systems. It defines the JIR task as a partially observable Markov decision process with explicit observations, states, actions, transitions, and rewards, and proposes an evaluation framework with Recall, Precision, R_relevance, and R_timeliness metrics. The authors construct JIR-Arena with 34 scenes (831 minutes) using a multi-entity, multi-turn simulation pipeline to approximate information needs, along with a three-layer content completion pipeline (IR, LLM checks, human verification) to generate high-quality references. A prototypical baseline system demonstrates that while foundation-model-based need inference is reasonably precise, retrieval quality and timeliness remain challenging, motivating further research and improvements. The work also provides fully released code and data to foster community development in this nascent but impactful area of proactive AI information services.

Abstract

Just-in-time Information Recommendation (JIR) is a service designed to deliver the most relevant information precisely when users need it, , addressing their knowledge gaps with minimal effort and boosting decision-making and efficiency in daily life. Advances in device-efficient deployment of foundation models and the growing use of intelligent wearable devices have made always-on JIR assistants feasible. However, there has been no systematic effort to formally define JIR tasks or establish evaluation frameworks. To bridge this gap, we present the first mathematical definition of JIR tasks and associated evaluation metrics. Additionally, we introduce JIR-Arena, a multimodal benchmark dataset featuring diverse, information-request-intensive scenarios to evaluate JIR systems across critical dimensions: i) accurately inferring user information needs, ii) delivering timely and relevant recommendations, and iii) avoiding irrelevant content that may distract users. Developing a JIR benchmark dataset poses challenges due to subjectivity in estimating user information needs and uncontrollable system variables affecting reproducibility. To address these, JIR-Arena: i) combines input from multiple humans and large AI models to approximate information need distributions; ii) assesses JIR quality through information retrieval outcomes using static knowledge base snapshots; and iii) employs a multi-turn, multi-entity validation framework to improve objectivity and generality. Furthermore, we implement a baseline JIR system capable of processing real-time information streams aligned with user inputs. Our evaluation of this baseline system on JIR-Arena indicates that while foundation model-based JIR systems simulate user needs with reasonable precision, they face challenges in recall and effective content retrieval. To support future research in this new area, we fully release our code and data.

Figures (8)

• Figure 1: The JIR system takes in user persona and multimodal data stream, and proactively outputs JIRs based on knowledge bases to satisfy the user's instant information needs.
• Figure 2: A usable form of a JIR instance, with $\langle t_s, q, Ref, t_e \rangle$ to be the minimally required field for measuring the JIR's quality, $p$ for characterizing the need distribution, and $\langle S, I \rangle$ for system display.
• Figure 3: Our pipeline of constructing the JIR-Arena benchmark dataset.
• Figure 4: Evaluation metrics of the JIR systems (top-right): Two global metrics, $Recall$ and $Precision$, based on alignment of ground truth and the candidate answer (top-left), and $R_{relevance}$ and $R_{timeliness}$, aggregated from the relevance and timeliness scores of each JIR instance (bottom).
• Figure 5: Example of labeled needs across scene 9Rxb2px3Qcl in addition to the generated needs by the language models.