Efficient and Accurate Memorable Conversation Model using DPO based on sLLM

TL;DR

The paper tackles memory management in multi-session dialogues for small, on-device LLMs by introducing a memory-augmented framework that separates memory maintenance from response generation. It leverages Direct Preference Optimization (DPO) to learn effective memory updates via negative sampling, enabling accurate reflection of prior conversations while keeping memory concise. Experimental results show that DPO-based training improves memory accuracy (e.g., higher BERTScore) and enhances dialog generation metrics (fluency, coherence, consistency) even when the base dialog model is smaller than baselines. The findings indicate that efficient memory management with DPO yields strong performance and resource efficiency, making it suitable for on-device, multi-session conversational agents.

Abstract

In multi-session dialog system, it is essential to continuously update the memory as the session progresses. Simply accumulating memory can make it difficult to focus on the content of the conversation for inference due to the limited input sentence size. Therefore, efficient and accurate conversation model that is capable of managing memory to reflect the conversation history continuously is necessary. This paper presents a conversation model that efficiently manages memory as sessions progress and incorporates this into the model to reflect the conversation history accurately with 3 methodologies: SFT, DPO and DPO with SFT model. Our model using DPO algorithm shows an improvement about 0.0591 of BERTScore in memory accuracy, and the rate of responses reflecting the memory increased as well. Also, response generation performance enhanced about 4.292 in fluency, 3.935 in coherence, and 2.896 in consistency. This paper describes a training method that yields better performance than models with more than twice the parameter size, even when the model size is smaller. Thus, our model demonstrates efficiency not only in terms of accuracy but also in resource utilization.

Figures (7)

• Figure 1: Example of problem in common LLMs
• Figure 2: A proposed framework for memory-augmented small language models
• Figure 3: Overview of the methodology of training models.
• Figure 4: Example prompt of Dialog Generation Module
• Figure 5: Example of generating DPO negative sample prompt