Multi-Hop Question Answering: When Can Humans Help, and Where do They Struggle?

TL;DR

The paper investigates how humans can effectively collaborate with AI in multi-hop QA by conducting a five-task study on the 2WikiMultiHopQA benchmark to dissect human performance across decomposition, retrieval, and synthesis. It finds that humans perform best on answer integration ($97.3\%$) and maintain strong accuracy on direct multi-hop and single-hop QA (around $80.2\%$ and $84.1\%$), but struggle to recognize when multiple steps are required ($67.9\%$) and to decompose complex queries ($78.2\%$). These results motivate hybrid QA workflows where AI handles complexity detection and adaptive retrieval while humans contribute decomposition and final synthesis, balancing speed, cost, and accuracy. Overall, the work highlights targeted human strengths and machine-guided supports as a path to more efficient, controllable retrieval-augmented QA.

Abstract

Multi-hop question answering is a challenging task for both large language models (LLMs) and humans, as it requires recognizing when multi-hop reasoning is needed, followed by reading comprehension, logical reasoning, and knowledge integration. To better understand how humans might collaborate effectively with AI, we evaluate the performance of crowd workers on these individual reasoning subtasks. We find that while humans excel at knowledge integration (97\% accuracy), they often fail to recognize when a question requires multi-hop reasoning (67\% accuracy). Participants perform reasonably well on both single-hop and multi-hop QA (84\% and 80\% accuracy, respectively), but frequently make semantic mistakes--for example, answering "when" an event happened when the question asked "where." These findings highlight the importance of designing AI systems that complement human strengths while compensating for common weaknesses.

Figures (1)

• Figure 1: Overview of the five human evaluation tasks designed to analyze how humans perform in multi-hop retrieval-augmented generation (RAG). Task 1 serves as a baseline, asking participants to directly answer complex multi-hop questions. Tasks 2–4 decompose the RAG pipeline into core steps—query decomposition, single-hop answering, and answer integration—requiring capabilities in reasoning, reading comprehension, and knowledge synthesis. After guiding the participants through the complete multi-hop RAG stages, Task 5 evaluates whether participants can recognize the complexity of a query (single-hop vs. multi-hop), a prerequisite for adaptive retrieval strategies. These tasks help identify which steps are most challenging for humans and where human intervention is most valuable, informing the design of collaborative RAG systems.