PK-ICR: Persona-Knowledge Interactive Context Retrieval for Grounded Dialogue

TL;DR

This work introduces PK-ICR, a framework for Persona and Knowledge Dual Context Identification that jointly grounds dialogue by leveraging persona and knowledge contexts through QA-informed prompts. It treats knowledge retrieval as zero-shot top-1 scoring and uses a subsequent persona scoring stage with augmented dialogue, enabling efficient cross-context adaptation and state-of-the-art performance on the pkchat2022 benchmark. A novel Null-positive Rank Test is proposed to quantify hard-negative effects of persona-augmented dialogue, providing a threshold-free, rank-based evaluation of grounding quality. Collectively, PK-ICR advances multi-context grounding in dialogue systems with practical gains in retrieval accuracy and a framework for evaluating hard negatives, with potential extensions to more grounding sources and generation tasks.

Abstract

Identifying relevant persona or knowledge for conversational systems is critical to grounded dialogue response generation. However, each grounding has been mostly researched in isolation with more practical multi-context dialogue tasks introduced in recent works. We define Persona and Knowledge Dual Context Identification as the task to identify persona and knowledge jointly for a given dialogue, which could be of elevated importance in complex multi-context dialogue settings. We develop a novel grounding retrieval method that utilizes all contexts of dialogue simultaneously. Our method requires less computational power via utilizing neural QA retrieval models. We further introduce our novel null-positive rank test which measures ranking performance on semantically dissimilar samples (i.e. hard negatives) in relation to data augmentation.

Figures (6)

• Figure 1: Illustration of dialogue component interactions regarding PK-ICR. In (1), dialogue is augmented with each persona to allow necessary interactions between persona and knowledge. (2) performs knowledge retrieval with (1). (3) performs precise persona scoring.
• Figure 2: Null-positive Rank Test (NRT). $P_o$, $P_{\textit{pos}}$, $P_{\textit{neg}_i}$ denote null-positive sample, positive and negative personas respectively. We omit augmentation $+ D$ in the figure for brevity. $r_{\textit{min}} = -1$ and $r_{\textit{max}} = +3$ in this figure. Arrows are possible positions for $P_o$. Numbers on the rightmost side are the null-positive adjusted rank values, being $0$ right below $P_{\textit{pos}}$ (example in Table \ref{['tab:nrt_samples']}).
• Figure 3: Analysis of null-positive rank data for $P_i + D$ & $K_{\textit{true}_j}$ cross-encoder model. Delta value is the change between Zero-shot model and Ours model in terms of sample count (left axis). Ratio value is delta value divided by sample count for Zero-shot, in % (right axis). We report movements of delta in correct directions for rank $-1$, $0$ and ranks with long distance $3$, $4$. Similar results for bi-encoder (Fig. \ref{['NullposCount-bi']}).
• Figure A1: Resulting QA cross-task adaptation prompt of persona & knowledge pair (eq. \ref{['QAFormEq']}). Question form is "{persona} {dialogue}" while answer is "{knowledge}".
• Figure E1: Persona threshold ablation experiments with $P_i$ & $K_{\textit{true}_j}$ cross-encoder model. We report persona accuracy. $p_{\textit{thres}}$ is defined in eq. \ref{['PersonaEq']}. Dotted line correspond to Zero-shot model, and solid line is our best model. We find visible peak at $0.55$ with our best model while Zero-shot model performance keeps increasing $> 0.8$.