Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems

Zhenpeng Su, Xing Wu, Wei Zhou, Guangyuan Ma, Songlin Hu

TL;DR

This work proposes Dial-MAE (Dialogue Contextual Masking Auto-Encoder), a straightforward yet effective post-training technique tailored for dense encoders in dialogue response selection that achieves better alignment between the features of the dialogue context and response.

Abstract

Dialogue response selection aims to select an appropriate response from several candidates based on a given user and system utterance history. Most existing works primarily focus on post-training and fine-tuning tailored for cross-encoders. However, there are no post-training methods tailored for dense encoders in dialogue response selection. We argue that when the current language model, based on dense dialogue systems (such as BERT), is employed as a dense encoder, it separately encodes dialogue context and response, leading to a struggle to achieve the alignment of both representations. Thus, we propose Dial-MAE (Dialogue Contextual Masking Auto-Encoder), a straightforward yet effective post-training technique tailored for dense encoders in dialogue response selection. Dial-MAE uses an asymmetric encoder-decoder architecture to compress the dialogue semantics into dense vectors, which achieves better alignment between the features of the dialogue context and response. Our experiments have demonstrated that Dial-MAE is highly effective, achieving state-of-the-art performance on two commonly evaluated benchmarks.

Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems

TL;DR

This work proposes Dial-MAE (Dialogue Contextual Masking Auto-Encoder), a straightforward yet effective post-training technique tailored for dense encoders in dialogue response selection that achieves better alignment between the features of the dialogue context and response.

Abstract

Dialogue response selection aims to select an appropriate response from several candidates based on a given user and system utterance history. Most existing works primarily focus on post-training and fine-tuning tailored for cross-encoders. However, there are no post-training methods tailored for dense encoders in dialogue response selection. We argue that when the current language model, based on dense dialogue systems (such as BERT), is employed as a dense encoder, it separately encodes dialogue context and response, leading to a struggle to achieve the alignment of both representations. Thus, we propose Dial-MAE (Dialogue Contextual Masking Auto-Encoder), a straightforward yet effective post-training technique tailored for dense encoders in dialogue response selection. Dial-MAE uses an asymmetric encoder-decoder architecture to compress the dialogue semantics into dense vectors, which achieves better alignment between the features of the dialogue context and response. Our experiments have demonstrated that Dial-MAE is highly effective, achieving state-of-the-art performance on two commonly evaluated benchmarks.
Paper Structure (23 sections, 9 equations, 4 figures, 6 tables)

This paper contains 23 sections, 9 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Dialogue Response Retrieval
  4. PLM-based Dialogue Response Retrieval
  5. Methodology
  6. Masked Language Model Pre-training
  7. Dial-MAE: Dialogue Contextual Masking Auto-Encoder
  8. Fine-tuning for dialogue response selection
  9. Experiment
  10. Datasets
  11. Evaluation Metric
  12. Implementation Details
  13. Post-training.
  14. Fine-tuning.
  15. Results and Discussions
  16. ...and 8 more sections

Figures (4)

  • Figure 1: The model design for Dial-MAE. The input of the encoder is the dialogue context, and its next response and dialogue context embedding output by the encoder is used as the input to the decoder.
  • Figure 2: We discard the decoder, initialize the context encoder and response encoder using the encoder part of Dial-MAE, and fine-tune using contrastive learning. At inference time, We use a dot product to measure similarity.
  • Figure 3: Fine-tuning schedules on the dev set of Ubuntu Corpus. A longer fine-tuning schedule gives a noticeable improvement. The performance of Dial-MAE is always better than BERT$_{+CL}$.
  • Figure 4: Impact of layer number on Ubuntu Corpus.