Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TRIM: Token Reduction and Inference Modeling for Cost-Effective Language Generation

Alfredo Garrachón Ruiz, Tomás de la Rosa, Daniel Borrajo

TL;DR

TRIM addresses the high inference costs of large language models by omitting easily-inferable, low-information words during generation and reconstructing the distilled text with a smaller model. It introduces a word-inferability ranking based on $ΔP$ and builds NaLDA, a 44,800-entry dataset across five generation tasks, to evaluate reconstruction performance. Experiments show average token savings around 18–19% with minimal degradation in semantic meaning, and demonstrate that small reconstruction models like T5 can effectively reconstruct full text, especially when paired with GPT-4o as the generator. The approach is language-agnostic and offers practical implications for scalable, cost-efficient generation in real-world deployments.

Abstract

The high inference cost of Large Language Models (LLMs) poses challenges, especially for tasks requiring lengthy outputs. However, natural language often contains redundancy, which presents an opportunity for optimization. We have observed that LLMs can generate distilled language (i.e., concise outputs that retain essential meaning) when prompted appropriately. We propose TRIM, a pipeline for saving computational cost in which the LLM omits a predefined set of semantically irrelevant and easily inferable words based on the context during inference. Then, a specifically trained smaller language model with lower inference cost reconstructs the distilled answer into the ideal answer. Our experiments show promising results, particularly on the proposed NaLDA evaluation dataset focused on the reconstruction task, with 19.4% saved tokens on average for GPT-4o and only a tiny decrease in evaluation metrics. This suggests that the approach can effectively balance efficiency and accuracy in language processing tasks.

TRIM: Token Reduction and Inference Modeling for Cost-Effective Language Generation

TL;DR

TRIM addresses the high inference costs of large language models by omitting easily-inferable, low-information words during generation and reconstructing the distilled text with a smaller model. It introduces a word-inferability ranking based on and builds NaLDA, a 44,800-entry dataset across five generation tasks, to evaluate reconstruction performance. Experiments show average token savings around 18–19% with minimal degradation in semantic meaning, and demonstrate that small reconstruction models like T5 can effectively reconstruct full text, especially when paired with GPT-4o as the generator. The approach is language-agnostic and offers practical implications for scalable, cost-efficient generation in real-world deployments.

Abstract

The high inference cost of Large Language Models (LLMs) poses challenges, especially for tasks requiring lengthy outputs. However, natural language often contains redundancy, which presents an opportunity for optimization. We have observed that LLMs can generate distilled language (i.e., concise outputs that retain essential meaning) when prompted appropriately. We propose TRIM, a pipeline for saving computational cost in which the LLM omits a predefined set of semantically irrelevant and easily inferable words based on the context during inference. Then, a specifically trained smaller language model with lower inference cost reconstructs the distilled answer into the ideal answer. Our experiments show promising results, particularly on the proposed NaLDA evaluation dataset focused on the reconstruction task, with 19.4% saved tokens on average for GPT-4o and only a tiny decrease in evaluation metrics. This suggests that the approach can effectively balance efficiency and accuracy in language processing tasks.

Paper Structure

This paper contains 25 sections, 4 equations, 5 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompt Compression
  4. Inference Optimizations
  5. Text Reconstruction
  6. Exploratory Research
  7. Semantic Relevance Of Words
  8. Cost-Saving Pipeline
  9. Datasets & Models
  10. Evaluation
  11. Metrics for Reconstruction Task
  12. Inferable Word Set
  13. Reconstruction Model
  14. Results across different task types
  15. Discussion
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Diagram of the proposed cost-saving pipeline with examples
  • Figure 2: Evaluation at different levels of saved tokens.
  • Figure 3: Comparison of metrics for GPT-4o as $M_G$ and T5 as $M_R$ across the 5 different task types of the NaLDA dataset.
  • Figure 4: Vanilla Answer Length vs Length Difference Between Answers
  • Figure 5: Visual comparison of the results for the reconstruction models in the NaLDA dataset using GPT-4o as $M_G$