DINT Transformer

TL;DR

Experimental results demonstrate that DINT Transformer excels in accuracy and robustness across various practical applications, such as long-context language modeling and key information retrieval, and position DINT Transformer as a highly effective and promising architecture.

Abstract

DIFF Transformer addresses the issue of irrelevant context interference by introducing a differential attention mechanism that enhances the robustness of local attention. However, it has two critical limitations: the lack of global context modeling, which is essential for identifying globally significant tokens, and numerical instability due to the absence of strict row normalization in the attention matrix. To overcome these challenges, we propose DINT Transformer, which extends DIFF Transformer by incorporating a differential-integral mechanism. By computing global importance scores and integrating them into the attention matrix, DINT Transformer improves its ability to capture global dependencies. Moreover, the unified parameter design enforces row-normalized attention matrices, improving numerical stability. Experimental results demonstrate that DINT Transformer excels in accuracy and robustness across various practical applications, such as long-context language modeling and key information retrieval. These results position DINT Transformer as a highly effective and promising architecture.

Figures (13)

• Figure 1: Transformer often over-attends to irrelevant context (i.e., attention noise). DINT Transformer not only eliminates noise but also strengthens the attention to globally important tokens, such as 'Newton' in the sentence.
• Figure 2: Multi-head DINT Attention. DIFF Attention matrix implements reducing attention noise, while the Integration Attention matrix enhances global attention.
• Figure 4: Multi-needle retrieval results in 64K length.
• Figure 5: TREC dataset (6 classes)
• Figure 6: TREC-fine dataset (50 classes)