Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DecompressionLM: Deterministic, Diagnostic, and Zero-Shot Concept Graph Extraction from Language Models

Zhaochen Hong, Jiaxuan You

TL;DR

DecompressionLM tackles the challenge of exhaustively uncovering what language models know by moving beyond pre-defined prompts and sequential, stateful probing. It combines deterministic Van der Corput low-discrepancy sampling with arithmetic decoding to generate independent, parallel sequences that reveal a domain-specific concept graph, including edges inferred from co-occurrence. The method exposes how model quantization shapes knowledge breadth, showing AWQ-4bit preserves or expands concept coverage while other 4-bit schemes can fragment the knowledge graph, often without being reflected in perplexity. Corpus-grounded validation demonstrates that a majority of extracted concepts correspond to real legal documents, and that higher benchmark scores correlate with lower hallucination rates, underscoring the practical value of concept coverage as a diagnostic dimension for compressed models. Overall, DecompressionLM provides a principled, scalable, and interpretable framework for evaluating knowledge diversity and factual grounding in LLMs, with implications for deployment and benchmarking beyond traditional perplexity metrics.

Abstract

Existing knowledge probing methods rely on pre-defined queries, limiting extraction to known concepts. We introduce DecompressionLM, a stateless framework for zero-shot concept graph extraction that discovers what language models encode without pre-specified queries or shared cross-sequence state. Our method targets three limitations of common decoding-based probing approaches: cross-sequence coupling that concentrates probability mass on high-frequency prefixes, competitive decoding effects that suppress long-tail concepts, and scalability constraints arising from sequential exploration. Using Van der Corput low-discrepancy sequences with arithmetic decoding, DecompressionLM enables deterministic, embarrassingly parallel generation without shared state across sequences. Across two model families and five quantization variants, we find that activation-aware quantization (AWQ-4bit) expands concept coverage by 30-170%, while uniform quantization (GPTQ-Int4) induces 71-86% coverage collapse -- divergent behaviors not reliably reflected by explanation-level perplexity. Corpus-based verification further reveals a 17-point hallucination gap between top- and bottom-ranked MMLU-Pro Law models. DecompressionLM establishes concept coverage as a complementary evaluation dimension for assessing knowledge breadth and factual grounding in compressed models useful for their deployment.

DecompressionLM: Deterministic, Diagnostic, and Zero-Shot Concept Graph Extraction from Language Models

TL;DR

DecompressionLM tackles the challenge of exhaustively uncovering what language models know by moving beyond pre-defined prompts and sequential, stateful probing. It combines deterministic Van der Corput low-discrepancy sampling with arithmetic decoding to generate independent, parallel sequences that reveal a domain-specific concept graph, including edges inferred from co-occurrence. The method exposes how model quantization shapes knowledge breadth, showing AWQ-4bit preserves or expands concept coverage while other 4-bit schemes can fragment the knowledge graph, often without being reflected in perplexity. Corpus-grounded validation demonstrates that a majority of extracted concepts correspond to real legal documents, and that higher benchmark scores correlate with lower hallucination rates, underscoring the practical value of concept coverage as a diagnostic dimension for compressed models. Overall, DecompressionLM provides a principled, scalable, and interpretable framework for evaluating knowledge diversity and factual grounding in LLMs, with implications for deployment and benchmarking beyond traditional perplexity metrics.

Abstract

Existing knowledge probing methods rely on pre-defined queries, limiting extraction to known concepts. We introduce DecompressionLM, a stateless framework for zero-shot concept graph extraction that discovers what language models encode without pre-specified queries or shared cross-sequence state. Our method targets three limitations of common decoding-based probing approaches: cross-sequence coupling that concentrates probability mass on high-frequency prefixes, competitive decoding effects that suppress long-tail concepts, and scalability constraints arising from sequential exploration. Using Van der Corput low-discrepancy sequences with arithmetic decoding, DecompressionLM enables deterministic, embarrassingly parallel generation without shared state across sequences. Across two model families and five quantization variants, we find that activation-aware quantization (AWQ-4bit) expands concept coverage by 30-170%, while uniform quantization (GPTQ-Int4) induces 71-86% coverage collapse -- divergent behaviors not reliably reflected by explanation-level perplexity. Corpus-based verification further reveals a 17-point hallucination gap between top- and bottom-ranked MMLU-Pro Law models. DecompressionLM establishes concept coverage as a complementary evaluation dimension for assessing knowledge breadth and factual grounding in compressed models useful for their deployment.
Paper Structure (56 sections, 9 equations, 1 figure, 10 tables, 1 algorithm)

This paper contains 56 sections, 9 equations, 1 figure, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Knowledge Probing in Language Models
  4. Decoding and Bias in Language Models
  5. Model Quantization for Language Models
  6. Method
  7. Problem Formulation
  8. Van der Corput Sampling for Arithmetic Coding
  9. Arithmetic Coding for Language Models.
  10. Van der Corput Sequences.
  11. Parallel Stateless Generation.
  12. Concept Extraction and Validation
  13. Concept Listing Prompt.
  14. Concept Normalization and Merging.
  15. Corpus-Grounded Validation.
  16. ...and 41 more sections

Figures (1)

  • Figure 1: Diagram of the DecompressionLM pipeline. Given a domain-specific prompt, the language model generates tokens sampled from its output distribution. Multiple parallel sequences yield diverse outputs from which we extract legal concepts (e.g., "Constitutional law", "Torts"). Concepts are normalized and then merged across sequences to construct a concept graph for analysis.