Context Retrieval via Normalized Contextual Latent Interaction for Conversational Agent

TL;DR

PK-NCLI introduces Normalized Contextual Latent Interaction to ground persona and external knowledge in conversational agents, achieving more accurate knowledge grounding and lower perplexity with improved training efficiency. By sharing embeddings and using low-level word-level interactions via NColBERT-inspired scoring, it reduces redundant LM calls and enables embedding caching for faster inference. The method demonstrates substantial gains over PK-FoCus on the FoCus dataset, with results showing significant improvements in language quality and grounding while maintaining persona grounding and enabling practical efficiency benefits. These findings underscore the importance of model choice and hyperparameter tuning for grounding components in real-time conversational AI systems.

Abstract

Conversational agents leveraging AI, particularly deep learning, are emerging in both academic research and real-world applications. However, these applications still face challenges, including disrespecting knowledge and facts, not personalizing to user preferences, and enormous demand for computational resources during training and inference. Recent research efforts have been focused on addressing these challenges from various aspects, including supplementing various types of auxiliary information to the conversational agents. However, existing methods are still not able to effectively and efficiently exploit relevant information from these auxiliary supplements to further unleash the power of the conversational agents and the language models they use. In this paper, we present a novel method, PK-NCLI, that is able to accurately and efficiently identify relevant auxiliary information to improve the quality of conversational responses by learning the relevance among persona, chat history, and knowledge background through low-level normalized contextual latent interaction. Our experimental results indicate that PK-NCLI outperforms the state-of-the-art method, PK-FoCus, by 47.80%/30.61%/24.14% in terms of perplexity, knowledge grounding, and training efficiency, respectively, and maintained the same level of persona grounding performance. We also provide a detailed analysis of how different factors, including language model choices and trade-offs on training weights, would affect the performance of PK-NCLI.

Figures (5)

• Figure 1: Network Architecture of PK-FoCusjang2022call
• Figure 2: Overview of PK-NCLI Model Architecture
• Figure 3: Demonstration of $\texttt{NColBERT}$
• Figure 4: Effectiveness of $\alpha$ on KG
• Figure 5: Effectiveness of $\gamma$ on $LM$