Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts

Zhuohan Gu, Jiayi Yao, Kuntai Du, Junchen Jiang

TL;DR

LLMSteer tackles long-context understanding in LLM inference without fine-tuning by post-hoc, query-independent attention steering. It leverages dual-prefix contextual rereading to modify and reuse KV caches, aligning attention toward tokens consistently deemed important across passes. The method achieves substantial efficiency gains by remaining compatible with prefix caching and by reducing runtime delays relative to AutoPASTA and larger models, while also improving generation quality across multiple datasets. This work demonstrates a practical path to narrow the performance gap between small and large LLMs without expensive fine-tuning.

Abstract

As large language models (LLMs) show impressive performance on complex tasks, they still struggle with longer contextual understanding and high computational costs. To balance efficiency and quality, we introduce LLMSteer, a fine-tuning-free framework that enhances LLMs through query-independent attention steering. Tested on popular LLMs and datasets, LLMSteer narrows the performance gap with baselines by 65.9% and reduces the runtime delay by up to 4.8x compared to recent attention steering methods.

LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts

TL;DR

LLMSteer tackles long-context understanding in LLM inference without fine-tuning by post-hoc, query-independent attention steering. It leverages dual-prefix contextual rereading to modify and reuse KV caches, aligning attention toward tokens consistently deemed important across passes. The method achieves substantial efficiency gains by remaining compatible with prefix caching and by reducing runtime delays relative to AutoPASTA and larger models, while also improving generation quality across multiple datasets. This work demonstrates a practical path to narrow the performance gap between small and large LLMs without expensive fine-tuning.

Abstract

As large language models (LLMs) show impressive performance on complex tasks, they still struggle with longer contextual understanding and high computational costs. To balance efficiency and quality, we introduce LLMSteer, a fine-tuning-free framework that enhances LLMs through query-independent attention steering. Tested on popular LLMs and datasets, LLMSteer narrows the performance gap with baselines by 65.9% and reduces the runtime delay by up to 4.8x compared to recent attention steering methods.

Paper Structure

This paper contains 11 sections, 4 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Post-hoc attention steering
  3. LLMSteer design
  4. Evaluation
  5. Setup
  6. Results
  7. Limitations and Future work
  8. Conclusion
  9. Appendix / supplemental material
  10. Prefix prompts
  11. LLMSteer Algorithm

Figures (3)

  • Figure 1: LLMSteer system in an LLM inference workflow processes the same context twice using different prefix prompts to generate key-value caches. It then reweights the selected tokens.
  • Figure 2: End-to-end Delay vs. Generation Quality. We assume KV cache is already in GPU memory when the request is being served.
  • Figure 3: The two prefix prompts used in LLMSteer to generate different key-value caches. By processing the same context with these varied prefixes, LLMSteer encourages the model to interpret the context differently in each pass.