KTRL+F: Knowledge-Augmented In-Document Search

TL;DR

Ktrl+F tackles the problem of knowledge-augmented in-document search by requiring real-time identification of all semantic targets while leveraging external knowledge. It introduces Knowledge-Augmented Phrase Retrieval, a retrieval-based approach that augments phrase embeddings with knowledge from sources like Wikipedia, achieving a favorable balance between latency and accuracy. A dedicated Ktrl+F dataset (512 queries over 98 documents) and tailored metrics (List EM F1, List Overlap F1, Robustness Score, ms/Q) support rigorous evaluation, with a human study confirming practical benefits in search efficiency. The work demonstrates that simple, knowledge-aware retrieval can outperform fully generative approaches in speed while maintaining high coverage, suggesting broad implications for efficient, knowledge-grounded information access in real-world tasks.

Abstract

We introduce a new problem KTRL+F, a knowledge-augmented in-document search task that necessitates real-time identification of all semantic targets within a document with the awareness of external sources through a single natural query. KTRL+F addresses following unique challenges for in-document search: 1)utilizing knowledge outside the document for extended use of additional information about targets, and 2) balancing between real-time applicability with the performance. We analyze various baselines in KTRL+F and find limitations of existing models, such as hallucinations, high latency, or difficulties in leveraging external knowledge. Therefore, we propose a Knowledge-Augmented Phrase Retrieval model that shows a promising balance between speed and performance by simply augmenting external knowledge in phrase embedding. We also conduct a user study to verify whether solving KTRL+F can enhance search experience for users. It demonstrates that even with our simple model, users can reduce the time for searching with less queries and reduced extra visits to other sources for collecting evidence. We encourage the research community to work on KTRL+F to enhance more efficient in-document information access.

Figures (8)

• Figure 1: Comparison between in-document search and Ktrl+F problem. In-document search accesses the information in documents by either lexical search (Ctrl+F, Regular expression) or semantic search (MRC). Lexical search suffers from finding semantically matching keywords, and semantic search does not consider external knowledge. Ktrl+F requires an efficient way to utilize external knowledge to find all semantic targets in real-time.
• Figure 2: Overview of Ktrl+F dataset construction pipeline. We utilize real news articles as input documents (Step 1), and automatically generate queries and targets using LLAMA (Step 2). To enhance the reliability of the identified targets, each entity is re-verified with external knowledge and finalized in (Step 3-1). Additionally, we use the MRC model to eliminate queries that do not meet the criteria outlined in req 2 (Step 3-2).
• Figure 3: Overview of Knowledge-Augmented Phrase Retrieval.
• Figure 4: A comparison of in-document search systems. Ktrl+F plugin outperforms other systems overall.
• Figure 5: Prompt for generating queries and targets