Efficient OpAmp Adaptation for Zoom Attention to Golden Contexts

TL;DR

This work tackles QA with noisy and long-context references by introducing OpAmp adaptation, a parameter-efficient mechanism that denoises Transformer attention using adapters inspired by operational amplifiers. The core idea combines differential and common-mode components to produce a refined attention matrix with a controllable CMRR $\mathcal{K} = A_d / A_c$, implemented via lightweight adapters to avoid training from scratch. Empirical results on NCFT with base models like Qwen2.5-72B show superior performance over current SOTA on noisy-context benchmarks across long-context, multi-hop, and noisy-RAG tasks, with notable gains in CoQA and LooGLE among others. The approach offers practical improvements for retrieval-augmented and long-context QA while keeping computational costs reasonable, albeit with a small increase in parameters and modest inference latency.

Abstract

Large language models (LLMs) have shown significant promise in question-answering (QA) tasks, particularly in retrieval-augmented generation (RAG) scenarios and long-context applications. However, their performance is hindered by noisy reference documents, which often distract from essential information. Despite fine-tuning efforts, Transformer-based architectures struggle to prioritize relevant content. This is evidenced by their tendency to allocate disproportionate attention to irrelevant or later-positioned documents. Recent work proposes the differential attention mechanism to address this issue, but this mechanism is limited by an unsuitable common-mode rejection ratio (CMRR) and high computational costs. Inspired by the operational amplifier (OpAmp), we propose the OpAmp adaptation to address these challenges, which is implemented with adapters efficiently. By integrating the adapter into pre-trained Transformer blocks, our approach enhances focus on the golden context without costly training from scratch. Empirical evaluations on noisy-context benchmarks reveal that our Qwen2.5-OpAmp-72B model, trained with our OpAmp adaptation, surpasses the performance of state-of-the-art LLMs, including DeepSeek-V3 and GPT-4o.

Figures (6)

• Figure 1: Normalized attention score. Transformers often miss the golden document in a noisy context.
• Figure 2: Qwen2.5-OpAmp-72B achieves the best average performance in various noisy-context benchmarks compared to current SOTA LLMs.
• Figure 3: The operational amplifier with two input voltages $V_{\text{in}}^{+}$ and $V_{\text{in}}^{-}$. The CMRR $\mathcal{K}$ is controlled by resistances $R_1$, $R_2$, $R_3$, $R_4$.
• Figure 4: Overview of the OpAmp adaptation using \ref{['eq:opampattn']} with adapters.
• Figure 5: Normalized attention score. Our OpAmp model demonstrates significant attention denoise capability compared to the base model and QLoRA model.