Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion

Shangyu Wu, Ying Xiong, Yufei Cui, Xue Liu, Buzhou Tang, Tei-Wei Kuo, Chun Jason Xue

TL;DR

ReFusion addresses the inefficiency of retrieval-augmented approaches in non-knowledge-intensive tasks by directly fusing retrieval representations into hidden states rather than concatenating long retrieval contexts. It introduces a computation-efficient framework with a retriever module, a retrieval fusion module featuring a learnable reranker and an ordered-mask scheme, and an adaptive retrieval integrator that searches for layer-wise best combinations via bi-level optimization. Across 15 NKI tasks, ReFusion delivers superior and more robust performance than strong baselines, while significantly reducing the computational burden compared to retrieval-concatenation methods. The work demonstrates a practical, scalable approach to integrating external knowledge into transformers, with potential for further gains through additional ranking schemes and hyperparameter tuning.

Abstract

Retrieval-based augmentations (RA) incorporating knowledge from an external database into language models have greatly succeeded in various knowledge-intensive (KI) tasks. However, integrating retrievals in non-knowledge-intensive (NKI) tasks is still challenging. Existing works focus on concatenating retrievals with inputs to improve model performance. Unfortunately, the use of retrieval concatenation-based augmentations causes an increase in the input length, substantially raising the computational demands of attention mechanisms. This paper proposes a new paradigm of RA named \textbf{ReFusion}, a computation-efficient Retrieval representation Fusion with bi-level optimization. Unlike previous works, ReFusion directly fuses the retrieval representations into the hidden states of models. Specifically, ReFusion leverages an adaptive retrieval integrator to seek the optimal combination of the proposed ranking schemes across different model layers. Experimental results demonstrate that the proposed ReFusion can achieve superior and robust performance in various NKI tasks.

ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion

TL;DR

ReFusion addresses the inefficiency of retrieval-augmented approaches in non-knowledge-intensive tasks by directly fusing retrieval representations into hidden states rather than concatenating long retrieval contexts. It introduces a computation-efficient framework with a retriever module, a retrieval fusion module featuring a learnable reranker and an ordered-mask scheme, and an adaptive retrieval integrator that searches for layer-wise best combinations via bi-level optimization. Across 15 NKI tasks, ReFusion delivers superior and more robust performance than strong baselines, while significantly reducing the computational burden compared to retrieval-concatenation methods. The work demonstrates a practical, scalable approach to integrating external knowledge into transformers, with potential for further gains through additional ranking schemes and hyperparameter tuning.

Abstract

Retrieval-based augmentations (RA) incorporating knowledge from an external database into language models have greatly succeeded in various knowledge-intensive (KI) tasks. However, integrating retrievals in non-knowledge-intensive (NKI) tasks is still challenging. Existing works focus on concatenating retrievals with inputs to improve model performance. Unfortunately, the use of retrieval concatenation-based augmentations causes an increase in the input length, substantially raising the computational demands of attention mechanisms. This paper proposes a new paradigm of RA named \textbf{ReFusion}, a computation-efficient Retrieval representation Fusion with bi-level optimization. Unlike previous works, ReFusion directly fuses the retrieval representations into the hidden states of models. Specifically, ReFusion leverages an adaptive retrieval integrator to seek the optimal combination of the proposed ranking schemes across different model layers. Experimental results demonstrate that the proposed ReFusion can achieve superior and robust performance in various NKI tasks.
Paper Structure (26 sections, 7 equations, 4 figures, 12 tables)

This paper contains 26 sections, 7 equations, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. ReFusion: A Computation-Efficient Retrieval Representation Fusion with Bilevel Optimization
  4. The Retriever Module
  5. The Retrieval Fusion Module
  6. Reranking the Retrievals
  7. Ordered Mask Over Retrieval Representations
  8. The Adaptive Retrieval Integrator
  9. Search Space
  10. Searching with Bi-Level Optimization
  11. Experiment
  12. Datasets and Experimental Setting
  13. Main Results
  14. Ablation Study
  15. Analysis
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Impact of the number of concatenated retrievals on inputs and its effect on model's accuracy and FLOPs. RC (acc) and RF-add (acc) refer to the accuracy of retrieval-concatenation-based augmentation (RC) and retrieval representation fusion with addition (RF-add). RC (flops) and RF-add (flops) refer to the FLOPs of RC and RF-add.
  • Figure 2: Comparisons between retrieval concatenation-based augmentation (RC) and retrieval representation fusion (RF).
  • Figure 3: The architecture of the proposed ReFusion and the detailed structure of proposed modules.
  • Figure 4: Two different ranking schemes used in the fusion module.