TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview

TL;DR

TREC iKAT 2023 introduces a track for personalized conversational information seeking using a Personal Text Knowledge Base (PTKB) to tailor interactions to user context. It defines three core tasks—Statement Ranking, Passage Ranking, and Response Generation—and evaluates them on 11 train and 25 test topics with a large ClueWeb22-B passage subset, using baselines and multiple evaluation metrics. Across 24 automatic and 3 manual runs from seven teams, results show that generate-then-ground (G→R→G) pipelines typically outperform retrieve-then-generate (R→G), illustrating the value of leveraging LLM internal knowledge before grounding with retrieved passages. The study also analyzes PTKB provenance and groundedness, revealing nuanced effects of personalization depth and topic difficulty, and highlights resources and methodologies that advance the state of persona-aware conversational search. Overall, iKAT demonstrates the feasibility and challenges of building CSA systems that adapt to user context and decisional tasks, providing benchmarks and baselines for future research.

Abstract

Conversational Information Seeking has evolved rapidly in the last few years with the development of Large Language Models providing the basis for interpreting and responding in a naturalistic manner to user requests. iKAT emphasizes the creation and research of conversational search agents that adapt responses based on the user's prior interactions and present context. This means that the same question might yield varied answers, contingent on the user's profile and preferences. The challenge lies in enabling Conversational Search Agents (CSA) to incorporate personalized context to effectively guide users through the relevant information to them. iKAT's first year attracted seven teams and a total of 24 runs. Most of the runs leveraged Large Language Models (LLMs) in their pipelines, with a few focusing on a generate-then-retrieve approach.

Figures (7)

• Figure 1: Two flowcharts representing different dialogues between a prospective student and an AI assistant on the topic of finding a suitable university for a master's degree in computer science. On the left, the conversation (PTKB 1) revolves around a student with a bachelor's degree from Tilburg University and work experience, who prefers to stay in the Netherlands. The dialogue suggests top Dutch universities and narrows down to the top three based on ranking. On the right, the second conversation (PTKB 2) involves a student who cannot tolerate cold temperatures below -12°C and is planning to move to Canada for a master's degree. The assistant provides options for top Canadian universities and further refines the suggestions to those with favorable weather conditions, eventually offering detailed information about the University of Toronto upon request. Each conversation flow is guided by the student's preferences, leading to tailored university recommendations.
• Figure 2: Number turns evaluated per dialogue in the final judgment pool vs. the maximum depth of each topic.
• Figure 3: Performance of all automatic runs in terms of nDCG@5 on the passage ranking task.
• Figure 5: nDCG@5 at varying conversation turn depths on the passage ranking task. We report the average across runs, median or better.
• Figure 6: nDCG@5 at varying conversation turn depths on the passage ranking task, for turns that depend on PTKB statements vs. those that do not. We report the average across runs, median or better.