Backtracing: Retrieving the Cause of the Query

TL;DR

The task of backtracing is introduced, in which systems retrieve the text segment that most likely caused a user query, and the results show that there is room for improvement on backtracing and it requires new retrieval approaches.

Abstract

Many online content portals allow users to ask questions to supplement their understanding (e.g., of lectures). While information retrieval (IR) systems may provide answers for such user queries, they do not directly assist content creators -- such as lecturers who want to improve their content -- identify segments that _caused_ a user to ask those questions. We introduce the task of backtracing, in which systems retrieve the text segment that most likely caused a user query. We formalize three real-world domains for which backtracing is important in improving content delivery and communication: understanding the cause of (a) student confusion in the Lecture domain, (b) reader curiosity in the News Article domain, and (c) user emotion in the Conversation domain. We evaluate the zero-shot performance of popular information retrieval methods and language modeling methods, including bi-encoder, re-ranking and likelihood-based methods and ChatGPT. While traditional IR systems retrieve semantically relevant information (e.g., details on "projection matrices" for a query "does projecting multiple times still lead to the same point?"), they often miss the causally relevant context (e.g., the lecturer states "projecting twice gets me the same answer as one projection"). Our results show that there is room for improvement on backtracing and it requires new retrieval approaches. We hope our benchmark serves to improve future retrieval systems for backtracing, spawning systems that refine content generation and identify linguistic triggers influencing user queries. Our code and data are open-sourced: https://github.com/rosewang2008/backtracing.

Figures (10)

• Figure 1: The task of backtracing takes a query and identifies the context that triggers this query. Identifying the cause of a query can be challenging because of the lack of explicit labeling, large corpus size, and domain expertise to understand both the query and corpus.
• Figure 2: Retrieving the correct triggering context can provide insight into how to better satisfy the user's needs and improve content delivery. We formalize three real-world domains for which backtracing is important in providing context on a user's query: (a) The Lecture domain where the objective is to retrieve the cause of student confusion; (b) The News Article domain where the objective is to retrieve the cause of reader curiosity; (c) The Conversation domain where the objective is to retrieve the cause of user emotion (e.g., anger). The user's query is shown in the gray box and the triggering context is the green-highlighted sentence. Popular retrieval systems such as dense retriever-based and re-ranker based systems retrieve incorrect contexts shown in red.
• Figure 3: Illustration of backtracing. The goal of backtracing is to identify the most likely sentence from the ordered corpus $X$ that caused the query $q$. One example is the Lecture domain where the corpus is a lecture transcript and the query is a student question. The lecturer only discusses about projecting twice and the student further extends that idea to something not raised in the lecture, namely into projecting a matrix an arbitrary $n$ times.
• Figure 4: Each dataset plot shows the query similarity to the ground truth cause sentence (GT), to the corpus sentence with maximal similarity (Max), and the difference between the maximal and ground-truth similarity sentences (Diff).
• Figure 5: Each row plot is a per-domain histogram of where the ground-truth cause sentence lies in the corpus document. The x-axis reports the location of the cause sentence; $0$ means the cause sentence is the first sentence and $1$ the last sentence. The y-axis reports the count of cause sentences at that location.