Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Concept than Document: Context Compression via AMR-based Conceptual Entropy

Kaize Shi, Xueyao Sun, Xiaohui Tao, Lin Li, Qika Lin, Guandong Xu

TL;DR

The paper tackles information overload in long-context LLMs by introducing an unsupervised AMR-based context compression that uses concept-level entropy to select semantically informative units. By constructing sentence-level AMR graphs, computing $H(v)$ for each concept, and distilling high-information concepts before reconstructing fluent text, the method preserves core semantics while reducing context length. Empirical results on PopQA and EntityQuestions show improved accuracy and substantial compression, especially for long contexts, with robust performance across diverse LLM backbones. This approach demonstrates that stable linguistic representations like AMR can meaningfully enhance context engineering and efficiency in retrieval-augmented reasoning systems.

Abstract

Large Language Models (LLMs) face information overload when handling long contexts, particularly in Retrieval-Augmented Generation (RAG) where extensive supporting documents often introduce redundant content. This issue not only weakens reasoning accuracy but also increases computational overhead. We propose an unsupervised context compression framework that exploits Abstract Meaning Representation (AMR) graphs to preserve semantically essential information while filtering out irrelevant text. By quantifying node-level entropy within AMR graphs, our method estimates the conceptual importance of each node, enabling the retention of core semantics. Specifically, we construct AMR graphs from raw contexts, compute the conceptual entropy of each node, and screen significant informative nodes to form a condensed and semantically focused context than raw documents. Experiments on the PopQA and EntityQuestions datasets show that our method outperforms vanilla and other baselines, achieving higher accuracy while substantially reducing context length. To the best of our knowledge, this is the first work introducing AMR-based conceptual entropy for context compression, demonstrating the potential of stable linguistic features in context engineering.

Concept than Document: Context Compression via AMR-based Conceptual Entropy

TL;DR

The paper tackles information overload in long-context LLMs by introducing an unsupervised AMR-based context compression that uses concept-level entropy to select semantically informative units. By constructing sentence-level AMR graphs, computing for each concept, and distilling high-information concepts before reconstructing fluent text, the method preserves core semantics while reducing context length. Empirical results on PopQA and EntityQuestions show improved accuracy and substantial compression, especially for long contexts, with robust performance across diverse LLM backbones. This approach demonstrates that stable linguistic representations like AMR can meaningfully enhance context engineering and efficiency in retrieval-augmented reasoning systems.

Abstract

Large Language Models (LLMs) face information overload when handling long contexts, particularly in Retrieval-Augmented Generation (RAG) where extensive supporting documents often introduce redundant content. This issue not only weakens reasoning accuracy but also increases computational overhead. We propose an unsupervised context compression framework that exploits Abstract Meaning Representation (AMR) graphs to preserve semantically essential information while filtering out irrelevant text. By quantifying node-level entropy within AMR graphs, our method estimates the conceptual importance of each node, enabling the retention of core semantics. Specifically, we construct AMR graphs from raw contexts, compute the conceptual entropy of each node, and screen significant informative nodes to form a condensed and semantically focused context than raw documents. Experiments on the PopQA and EntityQuestions datasets show that our method outperforms vanilla and other baselines, achieving higher accuracy while substantially reducing context length. To the best of our knowledge, this is the first work introducing AMR-based conceptual entropy for context compression, demonstrating the potential of stable linguistic features in context engineering.

Paper Structure

This paper contains 23 sections, 6 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Context Engineering
  4. AMR-enhanced Large Language Models
  5. Information Theory in LLMs
  6. Methodology
  7. Problem Formulation
  8. AMR Graph Construction
  9. Information Entropy Computation
  10. Concept Distillation
  11. Context Compression and Reconstruction
  12. Experiments
  13. Datasets and Implementation Details
  14. Baseline Methods
  15. Evaluation Metrics
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Long retrieved documents contain much irrelevant content; our method keeps only key AMR-based concepts to form a semantically focused context.
  • Figure 2: The conceptual entropy-based workflow converts the sparse context in raw supporting documents into condensed AMR-based concepts, forming a compact semantic representation for LLMs inference.
  • Figure 3: Comparison of token-level compression ratios across different context compression methods.