Hypernym Mercury: Token Optimization Through Semantic Field Constriction And Reconstruction From Hypernyms. A New Text Compression Method
Chris Forrester, Octavia Sulea
TL;DR
The paper addresses the rising cost of token usage in LLM prompts by introducing Mercury, a novel word-level semantic compression framework. It combines a new latent representation called a 'dart' with a Field Constrictor to restructure text into a concise core (via hypernyms) and a detailed appendix, followed by reconstruction using multiple models and a Shapley-value based importance evaluation. Key contributions include the dart representation, a lossless or near-lossless compression mechanism, and cross-model validation that maintains semantic fidelity across varying compression rates, establishing new benchmarks for text encoding in RAG systems. The approach promises practical impact by dramatically reducing prompt tokens without sacrificing saliency, enabling scalable, verifiable, and model-agnostic downstream reasoning in agentic AI applications.
Abstract
Compute optimization using token reduction of LLM prompts is an emerging task in the fields of NLP and next generation, agentic AI. In this white paper, we introduce a novel (patent pending) text representation scheme and a first-of-its-kind word-level semantic compression of paragraphs that can lead to over 90% token reduction, while retaining high semantic similarity to the source text. We explain how this novel compression technique can be lossless and how the detail granularity is controllable. We discuss benchmark results over open source data (i.e. Bram Stoker's Dracula available through Project Gutenberg) and show how our results hold at the paragraph level, across multiple genres and models.