Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M3: A Multi-Task Mixed-Objective Learning Framework for Open-Domain Multi-Hop Dense Sentence Retrieval

Yang Bai, Anthony Colas, Christan Grant, Daisy Zhe Wang

TL;DR

M3 is introduced, an advanced recursive Multi-hop dense sentence retrieval system built upon a novel Multi-task Mixed-objective approach for dense text representation learning that yields state-of-the-art performance on a large-scale open-domain fact verification benchmark dataset, FEVER.

Abstract

In recent research, contrastive learning has proven to be a highly effective method for representation learning and is widely used for dense retrieval. However, we identify that relying solely on contrastive learning can lead to suboptimal retrieval performance. On the other hand, despite many retrieval datasets supporting various learning objectives beyond contrastive learning, combining them efficiently in multi-task learning scenarios can be challenging. In this paper, we introduce M3, an advanced recursive Multi-hop dense sentence retrieval system built upon a novel Multi-task Mixed-objective approach for dense text representation learning, addressing the aforementioned challenges. Our approach yields state-of-the-art performance on a large-scale open-domain fact verification benchmark dataset, FEVER. Code and data are available at: https://github.com/TonyBY/M3

M3: A Multi-Task Mixed-Objective Learning Framework for Open-Domain Multi-Hop Dense Sentence Retrieval

TL;DR

M3 is introduced, an advanced recursive Multi-hop dense sentence retrieval system built upon a novel Multi-task Mixed-objective approach for dense text representation learning that yields state-of-the-art performance on a large-scale open-domain fact verification benchmark dataset, FEVER.

Abstract

In recent research, contrastive learning has proven to be a highly effective method for representation learning and is widely used for dense retrieval. However, we identify that relying solely on contrastive learning can lead to suboptimal retrieval performance. On the other hand, despite many retrieval datasets supporting various learning objectives beyond contrastive learning, combining them efficiently in multi-task learning scenarios can be challenging. In this paper, we introduce M3, an advanced recursive Multi-hop dense sentence retrieval system built upon a novel Multi-task Mixed-objective approach for dense text representation learning, addressing the aforementioned challenges. Our approach yields state-of-the-art performance on a large-scale open-domain fact verification benchmark dataset, FEVER. Code and data are available at: https://github.com/TonyBY/M3
Paper Structure (24 sections, 5 equations, 8 figures, 4 tables, 1 algorithm)

This paper contains 24 sections, 5 equations, 8 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Dense Text Retrieval
  4. Multi-hop Text Retrieval
  5. Method
  6. Overview
  7. Dense Sentence Retrieval
  8. Multi-task Learning
  9. Mixed-Objective Learning
  10. Sentence Reranking
  11. Hybrid Ranking
  12. Experimental Setup
  13. Results
  14. Evidence Retrieval
  15. End-to-end Fact Verification
  16. ...and 9 more sections

Figures (8)

  • Figure 1: A FEVER example where multi-hop sentence-level evidence from multiple Wikipedia documents is required for verification.
  • Figure 2: Canonical thee-stage fact verification framework.
  • Figure 3: M3 iterative dense sentence retrieval pipeline. DSR refers to the dense sentence retrieval model; SRR refers to the sentence reranking model; *-single and *-multi indicate whether the model is trained on single-hop or multi-hop examples. When no specific number of hops is given, the multi-hop retrieval process continues until the top-5 hybrid-ranked sentences stop changing.
  • Figure 4: M3-DSR multi-task learning framework. When $t=1$ (i.e., first-hop), the input of the query encoder is the original claim $c$.
  • Figure 5: M3-DSR mixed-objective learning framework. The same model is trained with different dataset-objective combinations sequentially.
  • ...and 3 more figures