Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-grained Conversational Decoding via Isotropic and Proximal Search

Yuxuan Yao, Han Wu, Qiling Xu, Linqi Song

TL;DR

This work addresses the limitations of generic decoding in dialogue by introducing isotropic and proximal search (IPS), a decoding-time objective that enforces locality within an utterance and isotropy across utterances. IPS decouples the influence of previously generated tokens from the dialogue context, using two signals: p_value_t (proximity) and i_value_t (isotropy), with h_RT as the average token representation; token selection from the top-m is guided by a weighted combination of model probability and the difference (p_value_t - i_value_t). Evaluations on DailyDialog and LCCC across automatic metrics (BERTScore, MAUVE, Distinct, G-Eval) and human judgments show that IPS yields more fluent, coherent, and human-like responses, with SimDRC+IPS delivering the strongest performance; IPS also maintains reasonable diversity. Overall, IPS advances dialogue decoding by producing semantically concentrated yet informative and discriminative responses, compatible with various backbones and decoding seeds, though with higher decoding time than traditional methods and room for speed improvements.

Abstract

General-purpose text decoding approaches are usually adopted for dialogue response generation. Although the quality of the generated responses can be improved with dialogue-specific encoding methods, conversational decoding methods are still under-explored. Inspired by \citet{wu2023learning} that a good dialogue feature space should follow the rules of locality and isotropy, we present a fine-grained conversational decoding method, termed \textit{isotropic and proximal search (IPS)}. Our method is designed to generate the semantic-concentrated response, while still maintaining informativeness and discrimination against the context. Experiments show that our approach outperforms existing decoding strategies in the dialogue field across both automatic and human evaluation metrics. More in-depth analyses further confirm the effectiveness of our approach.

Fine-grained Conversational Decoding via Isotropic and Proximal Search

TL;DR

This work addresses the limitations of generic decoding in dialogue by introducing isotropic and proximal search (IPS), a decoding-time objective that enforces locality within an utterance and isotropy across utterances. IPS decouples the influence of previously generated tokens from the dialogue context, using two signals: p_value_t (proximity) and i_value_t (isotropy), with h_RT as the average token representation; token selection from the top-m is guided by a weighted combination of model probability and the difference (p_value_t - i_value_t). Evaluations on DailyDialog and LCCC across automatic metrics (BERTScore, MAUVE, Distinct, G-Eval) and human judgments show that IPS yields more fluent, coherent, and human-like responses, with SimDRC+IPS delivering the strongest performance; IPS also maintains reasonable diversity. Overall, IPS advances dialogue decoding by producing semantically concentrated yet informative and discriminative responses, compatible with various backbones and decoding seeds, though with higher decoding time than traditional methods and room for speed improvements.

Abstract

General-purpose text decoding approaches are usually adopted for dialogue response generation. Although the quality of the generated responses can be improved with dialogue-specific encoding methods, conversational decoding methods are still under-explored. Inspired by \citet{wu2023learning} that a good dialogue feature space should follow the rules of locality and isotropy, we present a fine-grained conversational decoding method, termed \textit{isotropic and proximal search (IPS)}. Our method is designed to generate the semantic-concentrated response, while still maintaining informativeness and discrimination against the context. Experiments show that our approach outperforms existing decoding strategies in the dialogue field across both automatic and human evaluation metrics. More in-depth analyses further confirm the effectiveness of our approach.
Paper Structure (30 sections, 7 equations, 2 figures, 10 tables)

This paper contains 30 sections, 7 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminary
  4. Dialogue response generation
  5. Dialogue modeling
  6. Isotropic and Proximal Search
  7. Experiments
  8. Baselines
  9. Settings
  10. Evaluation Metrics
  11. Results and Discussion
  12. Ablation Study
  13. Hyperparameter Analysis
  14. Conclusion
  15. Appendix
  16. ...and 15 more sections

Figures (2)

  • Figure 1: Ablation study on the DailyDialog dataset.
  • Figure 2: An image of a cosine similarity heatmap