Towards the Holographic Characteristic of LLMs for Efficient Short-text Generation

TL;DR

The paper addresses the inefficiency of autoregressive LLMs in short-text generation by uncovering a Holographic Characteristic, where target-side keywords tend to surface early in generation. It proposes HOLO, a plugin that (i) extracts keywords from the first two decoding steps using a first-order Markov approximation $P(y_i|y_{<i},X)\approx P(y_i|y_{i-1},X)$ and a nucleus-like keyword subset $V_1^{(p)}$ with $p=0.9$, yielding $V_{\mathcal{F}}^y$, and (ii) completes sentences via a modified POINTER with mask-predict guided by keyword chains. Experiments across EVA2.0-2.8B, ChatGLM-6B, and Belle-13B on Chinese dialogue datasets show HOLO achieves comparable automatic and human-like quality to baselines while delivering substantial inference-time and memory savings, demonstrating the practical viability of a model-agnostic acceleration plugin. The work highlights that early-generation signals encode rich target-side semantics and that parallelized, lexically constrained generation can yield efficient, coherent short-text outputs with broad applicability. All mathematical notation in this summary is presented within $...$ delimiters, e.g., $P_{\mathcal{F}}(w|X)$ and $P(y_i|y_{<i},X)\approx P(y_i|y_{i-1},X)$.

Abstract

The recent advancements in Large Language Models (LLMs) have attracted interest in exploring their in-context learning abilities and chain-of-thought capabilities. However, there are few studies investigating the specific traits related to the powerful generation capacity of LLMs. This paper aims to delve into the generation characteristics exhibited by LLMs. Through our investigation, we have discovered that language models tend to capture target-side keywords at the beginning of the generation process. We name this phenomenon the Holographic Characteristic of language models. For the purpose of exploring this characteristic and further improving the inference efficiency of language models, we propose a plugin called HOLO, which leverages the Holographic Characteristic to extract target-side keywords from language models within a limited number of generation steps and complements the sentence with a parallel lexically constrained text generation method. To verify the effectiveness of HOLO, we conduct massive experiments on language models of varying architectures and scales in the short-text generation scenario. The results demonstrate that HOLO achieves comparable performance to the baselines in terms of both automatic and human-like evaluation metrics and highlight the potential of the Holographic Characteristic.