ConSearcher: Supporting Conversational Information Seeking in Online Communities with Member Personas

Abstract

Many people browse online communities to learn from others' experiences and opinions, e.g., for constructing travel plans. Conversational search powered by large language models (LLMs) could ease this information-seeking task, but it remains under-investigated within the online community. In this paper, we first conducted an exploratory study (N=10) that indicated the helpfulness of a classic conversational search tool and identified room for improvement. Then, we proposed ConSearcher, an LLM-powered tool with dynamically generated member personas based on user queries to facilitate conversational search in the community. In ConSearcher, users can clarify their interests by checking what a simulated member similar to them may ask and get responses from diverse members' perspectives. A within-subjects study (N=27) showed that compared to two conversational search baselines, ConSearcher led to significantly higher information-seeking outcome and user engagement but raised concerns about over-personalization. We discuss implications for supporting conversational information seeking in online communities.

Figures (7)

• Figure 1: BaseAgent used in the exploratory study: (A) community interface with posts, comments, and a search box; (B) BaseAgent that B1) answers user queries with links to relevant community content and B2) recommends follow-up questions based on dialog history; (C) pop-up menu that allows users to highlight selected content or prompt BaseAgent to summarize it.
• Figure 2: Demonstration of ConSearcher using the 'Japan travel' task as an example task. (A): ConSearcher in the community sidebar. (B): Factor Map displays factors relevant to user queries and the circle size reflects the current volume of posts. (C): Seeker Persona Panel with (D): a detailed seeker persona. (E): Provider Persona Panel.
• Figure 3: Computational workflow of ConSearcher's persona-driven conversational search. In the user study, BaseAgent does not have the modules of (A) seeker personas and (B) provider personas, while the BaseSearcher does not have the module of (B) provider personas.
• Figure 4: User study procedure. The tasks and systems (i.e., BaseAgent, BaseSearcher, ConSearcher) were counterbalanced using Latin Square, and a post-interview was conducted. The study lasted approximately 3 hours.
• Figure 5: The experiment's statistical results about BaseAgent, BaseSearcher and ConSearcher interface. All items are using one-way repeated measures ANOVA and Bonferroni post-hoc test are then conducted for pairwise comparisons. All items except the "Meaningful point number" for RQ1 is measured using a standard 7-point Likert scale (1 - strongly disagree/very light burden; 7 - strongly agree/very heavy burden). The confidence interval is 0.95. Note:$**:p<0.010;*:p<0.050;+:0.050<p<0.100$ ; within-subjects; N = 27.