Concise and Precise Context Compression for Tool-Using Language Models

TL;DR

This work tackles the efficiency bottleneck of tool-using large language models by compressing long tool documentation into concise, precise summaries. It introduces two strategies—Selective Compression, which preserves critical tool and parameter names as raw text, and Block Compression, which achieves variable-length compression by partitioning documentation into fixed-size blocks—then integrates them into a joint training framework. Across API-Bank and APIBench, the approach attains performance comparable to an uncompressed upper bound at up to $16\times$ compression, with selective compression reducing key-name errors and block compression enhancing basic soft compression. The result is a practical recipe for deploying tool-using LLMs with dramatically reduced context lengths while maintaining accuracy, albeit with upfront pre-training costs and reliance on task-specific priors.

Abstract

Through reading the documentation in the context, tool-using language models can dynamically extend their capability using external tools. The cost is that we have to input lengthy documentation every time the model needs to use the tool, occupying the input window as well as slowing down the decoding process. Given the progress in general-purpose compression, soft context compression is a suitable approach to alleviate the problem. However, when compressing tool documentation, existing methods suffer from the weaknesses of key information loss (specifically, tool/parameter name errors) and difficulty in adjusting the length of compressed sequences based on documentation lengths. To address these problems, we propose two strategies for compressing tool documentation into concise and precise summary sequences for tool-using language models. 1) Selective compression strategy mitigates key information loss by deliberately retaining key information as raw text tokens. 2) Block compression strategy involves dividing tool documentation into short chunks and then employing a fixed-length compression model to achieve variable-length compression. This strategy facilitates the flexible adjustment of the compression ratio. Results on API-Bank and APIBench show that our approach reaches a performance comparable to the upper-bound baseline under up to 16x compression ratio.

Figures (2)

• Figure 1: An example of tool documentation compression. Compared with the key information (red), the other content (green) is more verbose and suitable for compression into summary sequences (yellow).
• Figure 2: Overview of our method for tool documentation compression. When compressing a tool's documentation, we cut out the key information as key blocks (red) and chunk the rest into plain blocks (green). We use the concatenation of key blocks and compressed plain blocks (yellow) as the compressed documentation. We supervise the decoder output conditioned on compressed documentation to train the compressor and the decoder end-to-end.