RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering

TL;DR

RoMQA presents a robustness-focused benchmark for QA that requires reasoning over multiple evidence passages and returns many correct answers. It constructs clusters of related constraints from Wikidata, generates logical queries, and crowdsources natural-language questions to evaluate how models handle constraint variations. The study finds zero-shot and few-shot methods perform only marginally above naive baselines, while supervised retrieval improves results yet remains well below gold-evidence upper bounds, especially in open-ended settings. Overall, RoMQA reveals substantial gaps in robustness and provides a clear direction for developing more resilient, multi-evidence QA systems.

Abstract

We introduce RoMQA, the first benchmark for robust, multi-evidence, multi-answer question answering (QA). RoMQA contains clusters of questions that are derived from related constraints mined from the Wikidata knowledge graph. RoMQA evaluates robustness of QA models to varying constraints by measuring worst-case performance within each question cluster. Compared to prior QA datasets, RoMQA has more human-written questions that require reasoning over more evidence text and have, on average, many more correct answers. In addition, human annotators rate RoMQA questions as more natural or likely to be asked by people. We evaluate state-of-the-art large language models in zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is challenging: zero-shot and few-shot models perform similarly to naive baselines, while supervised retrieval methods perform well below gold evidence upper bounds. Moreover, existing models are not robust to variations in question constraints, but can be made more robust by tuning on clusters of related questions. Our results show that RoMQA is a challenging benchmark for large language models, and provides a quantifiable test to build more robust QA methods.

Figures (11)

• Figure 1: A cluster of related questions, implicit constraints, evidence text, and answers from RoMQA. Within a RoMQA cluster, related questions differ in implicit constraints. In addition to evaluating model performance across questions, RoMQA evaluates robustness to variations in question constraints by scoring worst-case performance among related questions.
• Figure 2: Dataset comparison over question, evidence, and answer size distributions.
• Figure 3: Question diversity as measured by the number of unique noun-phrases in 500 randomly sampled questions from the development set of each dataset. The batches are randomly sampled 4 times to compute standard deviation.
• Figure 4: The distribution of questions naturalness ratings by 3 annotators on 1,000 randomly sampled questions from the development set of each dataset. Each annotator rates four questions shuffled in random order, one from each dataset. he annotator is asked whether they would ask the question themselves, and if they think someone else would ask the question.
• Figure 5: Correlation with model performance ($\mathrm{F}_1$) on the closed setting. Imprecise questions with many answers are easier to answer (higher $\mathrm{F}_1$). Questions based on general propositions that co-occur with many different entities are easier to answer. Questions with more constraints are more difficult to answer.