Apollonion: Profile-centric Dialog Agent

TL;DR

Apollonion tackles user-perspective personalization in dialog agents by introducing a profile-centric framework that initializes and iteratively updates a structured user profile. The pipeline comprises Profile Initialization, Retrieve, Reflect, and Response, leveraging three retrieval modes (Embedding-based, LLM-based, Full Content) and a Reflect module to extract latent user traits from queries, thereby producing personalized responses via $R = ext{LLM}( ilde{P}, ilde{Ref}, ilde{H}, Q; ext{Prompt})$. It uses Meituan transaction data to build initial profiles and proposes offline evaluation protocols for Personalization (Profile Initialization, Retrieve, Reflect, Response) as scalable alternatives to human judgments. Experimental results indicate that LLM-based generation, especially GPT-4, yields more inductive and discriminative profiles and that retrieval and reflection components meaningfully enhance personalization. The work demonstrates practical pathways to deploy personalized dialog agents and provides a concrete, scalable methodology for evaluating personalization in real-world settings.

Abstract

The emergence of Large Language Models (LLMs) has innovated the development of dialog agents. Specially, a well-trained LLM, as a central process unit, is capable of providing fluent and reasonable response for user's request. Besides, auxiliary tools such as external knowledge retrieval, personalized character for vivid response, short/long-term memory for ultra long context management are developed, completing the usage experience for LLM-based dialog agents. However, the above-mentioned techniques does not solve the issue of \textbf{personalization from user perspective}: agents response in a same fashion to different users, without consideration of their features, such as habits, interests and past experience. In another words, current implementation of dialog agents fail in ``knowing the user''. The capacity of well-description and representation of user is under development. In this work, we proposed a framework for dialog agent to incorporate user profiling (initialization, update): user's query and response is analyzed and organized into a structural user profile, which is latter served to provide personal and more precise response. Besides, we proposed a series of evaluation protocols for personalization: to what extend the response is personal to the different users. The framework is named as \method{}, inspired by inscription of ``Know Yourself'' in the temple of Apollo (also known as \method{}) in Ancient Greek. Few works have been conducted on incorporating personalization into LLM, \method{} is a pioneer work on guiding LLM's response to meet individuation via the application of dialog agents, with a set of evaluation methods for measurement in personalization.

Figures (13)

• Figure 1: "Know yourself" is a philosophical maxim which was inscribed upon the Temple of Apollo (also known as Apollonion) in the ancient Greek precinct of Delphi. It becomes famous and survive today for it is believed that the great philosopher Socrate cite it as creed: The most important thing in life is to gain understanding of oneself.
• Figure 2: Overflow of Apollonion: Rounded box in blue represents modules using LLM or other deep learning models. Transparent Box represents intermediate results from modules. Embedding based and LLM retrieval is demonstrated in upper right corner. We provide evaluation protocols for measuring "Personalization", as illustrated in left up corner.
• Figure 3: User Profiling Procedures of Apollonion: (1) Meituan User Records Collection, (2) Behavior Sequence Integration, (3) User Profile Generation.
• Figure 4: Experiments workflow of Apollonion. Figure (a): Metrics of online performance. User experiences are reflected in dialog duration and etc. This metric is adopted after official launch and commonly used in commercial dialog agents. Figure (b): Offline indicator, especially "Personalization" focused in Apollonion. Performance of each modules are measured. Figure (c): Qualification of Personalization given profile and corresponding response. It is used as a base measurement for module evaluation in Figure (b).
• Figure 5: Ideal design v.s. Practical situation in personalization measurement. Dashed boxes (left side) represents ideal evaluation but infeasible due to inaccessible of true profile. Solid boxes (right side) illustrates the practical solution: Performance of Response and Retrieve is measured by personalization between response and initial profile. Performance of profile initialing is evaluated on User Prediction Task and Recommendation Task.

Theorems & Definitions (1)

• Definition 5.1: Properties of Personalization