Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond More Context: Retrieval Diversity Boosts Multi-Turn Intent Understanding

Zhiming Lin

TL;DR

This work addresses multi-turn dialogue state tracking under tight token budgets and questions whether diversity in retrieved exemplars improves LLM-based intent understanding beyond simply adding more tokens. It introduces LDRA, a diversity-aware retrieval framework that balances intent coverage and linguistic variety via a combined objective R(S)=αG(S)+(1−α)D(S) and constrains exemplar relevance and per-label quotas. LDRA is integrated with a structured instruction prompt and a calibrated LLM verifier, and its performance is validated on SGD and MultiWOZ 2.4, achieving state-of-the-art Joint Goal Accuracy under budget constrained prompting and showing strong data efficiency and cross-dataset generalization. The findings demonstrate that set-level content diversity—not merely longer prompts or exemplar order—drives practical gains, offering deployable guidance for budget-constrained multi-turn intent systems and paving the way for adaptive budget control and broader domain applicability.

Abstract

Multi turn intent understanding is central to task oriented chatbots, yet real deployments face tight token budgets and noisy contexts, and most retrieval pipelines emphasize relevance while overlooking set level diversity and confounds such as more context or exemplar order. We ask whether retrieval diversity, rather than longer prompts, systematically improves LLM intent understanding under fixed budgets. We present a diversity aware retrieval framework that selects in context exemplars to balance intent coverage and linguistic variety, and integrates this selection with standard LLM decoders; the evaluation enforces budget matched prompts and randomized positions, and includes sensitivity analyses over exemplar count, diversity strength, and backbone size. On MultiWOZ 2.4 and SGD, the approach achieves strong gains in Joint Goal Accuracy under equal token budgets, surpassing strong LLM/DST baselines, with consistent improvements across K from 4 to 7 and moderate latency. Overall, the study isolates and validates the impact of content diversity in retrieval and offers a simple, deployable selection principle for building accurate, budget constrained multi turn intent systems.

Beyond More Context: Retrieval Diversity Boosts Multi-Turn Intent Understanding

TL;DR

This work addresses multi-turn dialogue state tracking under tight token budgets and questions whether diversity in retrieved exemplars improves LLM-based intent understanding beyond simply adding more tokens. It introduces LDRA, a diversity-aware retrieval framework that balances intent coverage and linguistic variety via a combined objective R(S)=αG(S)+(1−α)D(S) and constrains exemplar relevance and per-label quotas. LDRA is integrated with a structured instruction prompt and a calibrated LLM verifier, and its performance is validated on SGD and MultiWOZ 2.4, achieving state-of-the-art Joint Goal Accuracy under budget constrained prompting and showing strong data efficiency and cross-dataset generalization. The findings demonstrate that set-level content diversity—not merely longer prompts or exemplar order—drives practical gains, offering deployable guidance for budget-constrained multi-turn intent systems and paving the way for adaptive budget control and broader domain applicability.

Abstract

Multi turn intent understanding is central to task oriented chatbots, yet real deployments face tight token budgets and noisy contexts, and most retrieval pipelines emphasize relevance while overlooking set level diversity and confounds such as more context or exemplar order. We ask whether retrieval diversity, rather than longer prompts, systematically improves LLM intent understanding under fixed budgets. We present a diversity aware retrieval framework that selects in context exemplars to balance intent coverage and linguistic variety, and integrates this selection with standard LLM decoders; the evaluation enforces budget matched prompts and randomized positions, and includes sensitivity analyses over exemplar count, diversity strength, and backbone size. On MultiWOZ 2.4 and SGD, the approach achieves strong gains in Joint Goal Accuracy under equal token budgets, surpassing strong LLM/DST baselines, with consistent improvements across K from 4 to 7 and moderate latency. Overall, the study isolates and validates the impact of content diversity in retrieval and offers a simple, deployable selection principle for building accurate, budget constrained multi turn intent systems.
Paper Structure (52 sections, 19 equations, 6 figures, 2 tables)

This paper contains 52 sections, 19 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi–turn DST: from encoders to LLM/ICL.
  4. Retrieval–augmented DST.
  5. Diversity-aware selection in IR.
  6. Method: LDRA (Linguistic-Diversity Retrieval-Augmentation)
  7. Problem Setting and Notation
  8. Context-Aware Query Encoding
  9. Similarity Retrieval
  10. Diversity-Aware Re-ranking
  11. Label diversity.
  12. Text diversity.
  13. Combined objective (R-score).
  14. Constrained subset selection.
  15. Fast greedy selection with closed-form increments.
  16. ...and 37 more sections

Figures (6)

  • Figure 1: LDRA pipeline for multi-turn intent classification. (Left) An illustrative multi-turn conversation provides history $C_{1:n-1}$ and the current utterance $q_n$. The context encoder $f_\theta$ yields a context-aware query vector $z_n$. (Top-middle) A hybrid retriever queries the intent memory $\mathcal{M}=\{(x_i,y_i,e_i)\}$ to form a candidate pool $S_L$ using $\mathrm{Rel}(i\!\mid\!z_n)=\lambda_{\mathrm{vec}}\,s(z_n,e_i)+(1-\lambda_{\mathrm{vec}})\,\mathrm{BM25}(q_n,x_i)$. (Middle) A diversity engine re-ranks candidates by maximizing $R(S)=\alpha\,G(S)+(1-\alpha)\,D(S)$ with $G(S)=1-\sum_k p_k^2$ (label coverage) and $D(S)=1-\bar{s}(S)$ (linguistic variety), under constraints $s(z_n,e_i)\!\ge\!\tau$ and $n_k\!\le\!U$, producing a diverse set $T_K$ (“Retrieval Modified”: dedup + stable re-rank). (Bottom) The dialogue history is summarized into $\sigma(C_{1:n-1})$; together with $T_K$ it forms the structured prompt $\Pi=[\text{Instr};\,\sigma(C);\,q_n; \,T_K;\,\text{Answer}]$. The LLM verifier $\Phi$ outputs calibrated scores $S_\Phi(y\!\mid\!\Pi)$, and the inference step predicts $\hat{y}=\arg\max_y S_\Phi(y\!\mid\!\Pi)$ (example intent: Request_Refund).
  • Figure 2: Parameter sensitivity on MultiWOZ 2.4.(a) Effect of exemplar count $K$: JGA (bars) and AGA (line); JGA peaks at $K{=}7$, after which prompt redundancy/length hurts. (b) Effect of $\alpha$ with $K{=}7$: a balanced mixture is best; $\alpha{=}0.25$ attains the highest JGA. (c) Qwen3 scaling: larger backbones generally help; LDRA benefits across scales (0.6B$\rightarrow$32B).
  • Figure 3: Overall structure ablation on MultiWOZ 2.4. Bars show JGA; the curve shows AGA. Starting from the base model (Qwen3-14B), adding a task instruction yields a strong gain; similarity-only RAG and MMR bring incremental improvements; LDRA (ours) further increases JGA by promoting label coverage and linguistic variety in retrieved exemplars.
  • Figure 4: Diversity vs. classic re-ranking under equal token budgets.(Up) JGA vs. $K\in\{1,3,5,7,10\}$: LDRA (full) dominates and peaks at $K{=}7$; MMR/FPS improve over similarity Top-$K$. (Down)$R$-score vs. JGA across methods and $K$ shows a strong positive trend, indicating that sets with higher label coverage and linguistic variety translate into higher accuracy.
  • Figure 5: Effect of $\alpha$ (MultiWOZ 2.4).(Up) Heatmap of JGA across $\alpha\times K$ (token budget and $K$ matched across methods). JGA peaks for a balanced mixture (e.g., $\alpha\!\approx\!0.25$–$0.75$) and saturates or declines at extremes ($\alpha{=}0$/$1$). (Down) Radar profiles at $K{=}7$ contrasting $\{\text{JGA},\,G,\,1{-}\mathrm{Sim},\,R\}$ over $\alpha$ (metrics normalized to $[0,1]$ for display; $R$ is computed as $R=\alpha G+(1-\alpha)(1-\mathrm{Sim})$.
  • ...and 1 more figures