MDCR: A Dataset for Multi-Document Conditional Reasoning

TL;DR

MDCR tackles multi-document conditional reasoning with optimization by introducing a dataset that spans scholarships and jobs, and by formalizing a task where models must reason across documents to determine eligibility and optimal outcomes. The approach combines structured condition extraction, cross-document relationship analysis, and symbolic boolean solving to generate gold answers and evaluate LLMs under varied prompting strategies. Key findings show that current large language models struggle significantly with multi-document conditional reasoning, especially for optimization tasks, though providing gold information about conditions and relations yields substantial performance gains. The dataset and benchmark provide a realistic, high-difficulty testbed that highlights gaps in current capabilities and offers a path for future improvements in cross-document reasoning and optimization under uncertainty.

Abstract

The same real-life questions posed to different individuals may lead to different answers based on their unique situations. For instance, whether a student is eligible for a scholarship depends on eligibility conditions, such as major or degree required. ConditionalQA was proposed to evaluate models' capability of reading a document and answering eligibility questions, considering unmentioned conditions. However, it is limited to questions on single documents, neglecting harder cases that may require cross-document reasoning and optimization, for example, "What is the maximum number of scholarships attainable?" Such questions over multiple documents are not only more challenging due to more context having to understand, but also because the model has to (1) explore all possible combinations of unmentioned conditions and (2) understand the relationship between conditions across documents, to reason about the optimal outcome. To evaluate models' capability of answering such questions, we propose a new dataset MDCR, which can reflect real-world challenges and serve as a new test bed for complex conditional reasoning that requires optimization. We evaluate this dataset using the most recent LLMs and demonstrate their limitations in solving this task. We believe this dataset will facilitate future research in answering optimization questions with unknown conditions.

Figures (6)

• Figure 1: An example of documents and relationships.
• Figure 2: An example of inputs, outputs and reasoning with unmentioned conditions for optimization.
• Figure 3: Interface for condition extraction.
• Figure 4: Interface for identifying mapping and boolean relationships.
• Figure 5: Interface for labeling condition relationships.