Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CORG: Generating Answers from Complex, Interrelated Contexts

Hyunji Lee, Franck Dernoncourt, Trung Bui, Seunghyun Yoon

TL;DR

CORG introduces a graph-based Context Organizer to tackle complex interrelated contexts found in real-world corpora, where four relation types (distracting, ambiguous, counterfactual, duplicated) co-occur. It decomposes the problem into a graph constructor, a reranker, and an aggregator to group related contexts and generate comprehensive, cited answers with efficient inference. Through extensive experiments across eight models and two multi-factor datasets (AmbigDocs+ and ConflictQA+), CORG consistently outperforms grouping baselines and achieves competitive results with single-context approaches while reducing computational costs. The framework is retrieval-agnostic, training-free, and designed to improve entity disambiguation and answer recall, offering practical benefits for search, QA, and AI-assisted information synthesis in complex real-world knowledge bases.

Abstract

In a real-world corpus, knowledge frequently recurs across documents but often contains inconsistencies due to ambiguous naming, outdated information, or errors, leading to complex interrelationships between contexts. Previous research has shown that language models struggle with these complexities, typically focusing on single factors in isolation. We classify these relationships into four types: distracting, ambiguous, counterfactual, and duplicated. Our analysis reveals that no single approach effectively addresses all these interrelationships simultaneously. Therefore, we introduce Context Organizer (CORG), a framework that organizes multiple contexts into independently processed groups. This design allows the model to efficiently find all relevant answers while ensuring disambiguation. CORG consists of three key components: a graph constructor, a reranker, and an aggregator. Our results demonstrate that CORG balances performance and efficiency effectively, outperforming existing grouping methods and achieving comparable results to more computationally intensive, single-context approaches.

CORG: Generating Answers from Complex, Interrelated Contexts

TL;DR

CORG introduces a graph-based Context Organizer to tackle complex interrelated contexts found in real-world corpora, where four relation types (distracting, ambiguous, counterfactual, duplicated) co-occur. It decomposes the problem into a graph constructor, a reranker, and an aggregator to group related contexts and generate comprehensive, cited answers with efficient inference. Through extensive experiments across eight models and two multi-factor datasets (AmbigDocs+ and ConflictQA+), CORG consistently outperforms grouping baselines and achieves competitive results with single-context approaches while reducing computational costs. The framework is retrieval-agnostic, training-free, and designed to improve entity disambiguation and answer recall, offering practical benefits for search, QA, and AI-assisted information synthesis in complex real-world knowledge bases.

Abstract

In a real-world corpus, knowledge frequently recurs across documents but often contains inconsistencies due to ambiguous naming, outdated information, or errors, leading to complex interrelationships between contexts. Previous research has shown that language models struggle with these complexities, typically focusing on single factors in isolation. We classify these relationships into four types: distracting, ambiguous, counterfactual, and duplicated. Our analysis reveals that no single approach effectively addresses all these interrelationships simultaneously. Therefore, we introduce Context Organizer (CORG), a framework that organizes multiple contexts into independently processed groups. This design allows the model to efficiently find all relevant answers while ensuring disambiguation. CORG consists of three key components: a graph constructor, a reranker, and an aggregator. Our results demonstrate that CORG balances performance and efficiency effectively, outperforming existing grouping methods and achieving comparable results to more computationally intensive, single-context approaches.
Paper Structure (44 sections, 9 figures, 12 tables, 1 algorithm)

This paper contains 44 sections, 9 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Complex, Interrelated Contexts
  3. Definition of Relationships
  4. Statistics of Real-World Corpora
  5. Corpus Construction
  6. Evaluation Metric
  7. Analyzing Solution for each Factor
  8. Affect of each factor
  9. Solution for Each Factors individually
  10. Distracting Context
  11. Ambiguous Context
  12. Counterfactual Contexts
  13. Duplicated Contexts
  14. Context organizer
  15. Graph Constructor
  16. ...and 29 more sections

Figures (9)

  • Figure 1: In real-world corpora, contexts often exhibit complex interrelationships, which we classify into four categories: distracting, counterfactual, duplicated, and ambiguous.
  • Figure 2: Answer recall as the number of contexts increases for each factor. For ambiguous cases, note that since ambiguity can only exist between two contexts, sets with three or more contexts (x > 2) also include distracting relationships.
  • Figure 3: Overview of context organizer (COrg), composed of three components: graph constructor, reranker, and aggregator. Given a corpus with multiple relevant contexts and a question, the graph constructor and the reranker organize the corpus based on the question, and the aggregator generates a response containing all possible answers with references.
  • Figure 4: Trade-off between efficiency and performance for Llama2-7B in AmbigDocs+. Light color indicates better balance between the two; COrg shows the best followed by Summarize.
  • Figure 5: Instruction to Human Evaluators
  • ...and 4 more figures