Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dual-Density Inference for Efficient Language Model Reasoning

Zhengyi Zhao, Shubo Zhang, Yuxi Zhang, Huimin Wang, Binyang Li, Kam-Fai Wong

TL;DR

This paper identifies an efficiency gap in current LLM reasoning where internal computations rely on verbose natural language that hinders performance. It proposes Denser, a dual-density inference framework that uses high-density symbolic notation for intermediate reasoning and low-density natural language for final explanations, guided by a query-analysis module. Through extensive experiments across math, logic, coding, and general QA, Denser achieves substantial token reductions (up to 62%) while maintaining or improving accuracy and reducing latency, especially on multi-step problems. The approach is instantiated with domain-specific compression operators and density-aware prompting, and its effectiveness is demonstrated via ablations and case studies. Overall, the work shows that aligning representation density with the structural needs of reasoning tasks can yield tangible efficiency gains without sacrificing performance.

Abstract

Large Language Models (LLMs) have shown impressive capabilities in complex reasoning tasks. However, current approaches employ uniform language density for both intermediate reasoning and final answers, leading to computational inefficiency. Our observation found that reasoning process serves a computational function for the model itself, while answering serves a communicative function for human understanding. This distinction enables the use of compressed, symbol-rich language for intermediate computations while maintaining human-readable final explanations. To address this inefficiency, we present Denser: \underline{D}ual-d\underline{ens}ity inf\underline{er}ence, a novel framework that optimizes information density separately for reasoning and answering phases. Our framework implements this through three components: a query processing module that analyzes input problems, a high-density compressed reasoning mechanism for efficient intermediate computations, and an answer generation component that translates compressed reasoning into human-readable solutions. Experimental evaluation across multiple reasoning question answering benchmarks demonstrates that Denser reduces token consumption by up to 62\% compared to standard Chain-of-Thought methods while preserving or improving accuracy. These efficiency gains are particularly significant for complex multi-step reasoning problems where traditional methods generate extensive explanations.

Dual-Density Inference for Efficient Language Model Reasoning

TL;DR

This paper identifies an efficiency gap in current LLM reasoning where internal computations rely on verbose natural language that hinders performance. It proposes Denser, a dual-density inference framework that uses high-density symbolic notation for intermediate reasoning and low-density natural language for final explanations, guided by a query-analysis module. Through extensive experiments across math, logic, coding, and general QA, Denser achieves substantial token reductions (up to 62%) while maintaining or improving accuracy and reducing latency, especially on multi-step problems. The approach is instantiated with domain-specific compression operators and density-aware prompting, and its effectiveness is demonstrated via ablations and case studies. Overall, the work shows that aligning representation density with the structural needs of reasoning tasks can yield tangible efficiency gains without sacrificing performance.

Abstract

Large Language Models (LLMs) have shown impressive capabilities in complex reasoning tasks. However, current approaches employ uniform language density for both intermediate reasoning and final answers, leading to computational inefficiency. Our observation found that reasoning process serves a computational function for the model itself, while answering serves a communicative function for human understanding. This distinction enables the use of compressed, symbol-rich language for intermediate computations while maintaining human-readable final explanations. To address this inefficiency, we present Denser: \underline{D}ual-d\underline{ens}ity inf\underline{er}ence, a novel framework that optimizes information density separately for reasoning and answering phases. Our framework implements this through three components: a query processing module that analyzes input problems, a high-density compressed reasoning mechanism for efficient intermediate computations, and an answer generation component that translates compressed reasoning into human-readable solutions. Experimental evaluation across multiple reasoning question answering benchmarks demonstrates that Denser reduces token consumption by up to 62\% compared to standard Chain-of-Thought methods while preserving or improving accuracy. These efficiency gains are particularly significant for complex multi-step reasoning problems where traditional methods generate extensive explanations.

Paper Structure

This paper contains 46 sections, 14 equations, 4 figures, 32 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning Strategies.
  4. Efficient Reasoning.
  5. Empirical Analysis of Information Density in LLM Reasoning
  6. Information Density Patterns in LLM Reasoning
  7. Structural Analysis of Reasoning Language
  8. Denser: Dual-Density Inference
  9. Problem Formulation
  10. Framework Components
  11. Query Analysis Module.
  12. High-density Reasoning.
  13. Low-density Answering.
  14. Experiments
  15. Experimental Setup
  16. ...and 31 more sections

Figures (4)

  • Figure 1: Information Density Throughout Reasoning Process which shows information density throughout reasoning processing.
  • Figure 2: Token consumption and computational efficiency. Left: Average tokens consumed per problem by different methods across domains. Right: Computational efficiency (accuracy-to-token ratio) normalized to CoT baseline (higher is better). Denser achieves superior efficiency across all domains.
  • Figure 3: Scaling behavior with problem complexity. Red, Green, and Blue denotes CoT, SV and Denser, respectively. Left y-axis denotes token consumption and right y-axis means accuracy improvement.
  • Figure 4: Performance across model sizes.