A$^2$Search: Ambiguity-Aware Question Answering with Reinforcement Learning

TL;DR

A$^2$Search tackles the pervasive ambiguity in open-domain QA by introducing an annotation-free, end-to-end RL framework that detects questions with multiple valid answers, mines alternative answers through trajectory sampling with a search tool, and verifies them with multiple verifiers. Training uses Group Relative Policy Optimization with an AnsF1-based reward, enabling the model to produce multiple correct answers when warranted. The approach achieves state-of-the-art results across eight open-domain QA benchmarks with a single rollout and generalizes to AmbigQA, demonstrating robustness, efficiency, and practical value for reliable QA systems. This work highlights that explicitly modeling and leveraging ambiguity can substantially improve both the accuracy and reliability of QA systems in real-world settings.

Abstract

Recent advances in Large Language Models (LLMs) and Reinforcement Learning (RL) have led to strong performance in open-domain question answering (QA). However, existing models still struggle with questions that admit multiple valid answers. Standard QA benchmarks, which typically assume a single gold answer, overlook this reality and thus produce inappropriate training signals. Existing attempts to handle ambiguity often rely on costly manual annotation, which is difficult to scale to multi-hop datasets such as HotpotQA and MuSiQue. In this paper, we present A$^2$Search, an annotation-free, end-to-end training framework to recognize and handle ambiguity. At its core is an automated pipeline that detects ambiguous questions and gathers alternative answers via trajectory sampling and evidence verification. The model is then optimized with RL using a carefully designed $\mathrm{AnsF1}$ reward, which naturally accommodates multiple answers. Experiments on eight open-domain QA benchmarks demonstrate that A$^2$Search achieves new state-of-the-art performance. With only a single rollout, A$^2$Search-7B yields an average $\mathrm{AnsF1}@1$ score of $48.4\%$ across four multi-hop benchmarks, outperforming all strong baselines, including the substantially larger ReSearch-32B ($46.2\%$). Extensive analyses further show that A$^2$Search resolves ambiguity and generalizes across benchmarks, highlighting that embracing ambiguity is essential for building more reliable QA systems. Our code, data, and model weights can be found at https://github.com/zfj1998/A2Search

Figures (7)

• Figure 1: Rollout examples on an ambiguous question from MuSiQue. ReSearch yields different answers across rollouts, some diverging from the reference yet still evidence-supported, whereas A$^2$Search explicitly resolves ambiguity by retrieving multiple answers within a single rollout.
• Figure 2: Our pipeline for automatically identifying alternative answers in ambiguous questions.
• Figure 3: Answer count distribution in the final dataset.
• Figure 4: Training dynamics of A$^2$Search. Curves are smoothed for readability.
• Figure 5: Effect of entropy control on validation performance and the model’s entropy value.