Augmenting Researchy Questions with Sub-question Judgments

TL;DR

The paper addresses the lack of sub-question relevance judgments in the Researchy Questions dataset, which hampers training retrieval systems for complex information needs. It proposes automatic sub-question judgments using a large language model (Llama3.3 70B) guided by CRUX-inspired graded prompts, and constructs a candidate document pool via BM25 and a neural re-ranker (Qwen3 0.6B). The authors generate a large-scale, 0-5 relevance labeling of sub-questions across ~1.29 million sub-questions, producing over 24 million judgments to support training for factoid retrieval and QA. This resource enables improved training signals for retrieval models handling multi-faceted queries and serves as a community-improved benchmark for complex information-seeking tasks.

Abstract

The Researchy Questions dataset provides about 100k question queries with complex information needs that require retrieving information about several aspects of a topic. Each query in ResearchyQuestions is associated with sub-questions that were produced by prompting GPT-4. While ResearchyQuestions contains labels indicating what documents were clicked after issuing the query, there are no associations in the dataset between sub-questions and relevant documents. In this work, we augment the Researchy Questions dataset with LLM-judged labels for each sub-question using a Llama3.3 70B model. We intend these sub-question labels to serve as a resource for training retrieval models that better support complex information needs.

Figures (2)

• Figure 1: The prompt used to rerank documents with the pointwise Qwen3 0.6B reranking model.
• Figure 2: The prompt used to assess the relevance of a document (context) c to a sub-question q on a scale from 0 to 5, following prior work obtaining sub-question judgments ju2025controlled.