Exploring Rewriting Approaches for Different Conversational Tasks

TL;DR

The paper tackles whether a single LLM-based query rewriting module can be universally effective across diverse conversational tasks or should be specialized per use case. It introduces a parameterized framework that yields two main strategies—query rewrite (QR) and query fusion—applied to text-based Q&A and text-to-visualization generation, including short and long conversations. Through new datasets and rigorous evaluation using cosine similarity and BERT F1, the authors show that QR is superior for conversational QA while fusion better supports data-analytic visualization tasks, with fusion capable of summarizing arbitrarily long histories via compact rewritten prompts. They also explore a rewrite-necessity classifier to further refine when rewriting is applied. The results offer practical guidance for building adaptable, task-aware conversational assistants and highlight the value of dataset creation to assess rewriting strategies across modalities and conversation lengths.

Abstract

Conversational assistants often require a question rewriting algorithm that leverages a subset of past interactions to provide a more meaningful (accurate) answer to the user's question or request. However, the exact rewriting approach may often depend on the use case and application-specific tasks supported by the conversational assistant, among other constraints. In this paper, we systematically investigate two different approaches, denoted as rewriting and fusion, on two fundamentally different generation tasks, including a text-to-text generation task and a multimodal generative task that takes as input text and generates a visualization or data table that answers the user's question. Our results indicate that the specific rewriting or fusion approach highly depends on the underlying use case and generative task. In particular, we find that for a conversational question-answering assistant, the query rewriting approach performs best, whereas for a data analysis assistant that generates visualizations and data tables based on the user's conversation with the assistant, the fusion approach works best. Notably, we explore two datasets for the data analysis assistant use case, for short and long conversations, and we find that query fusion always performs better, whereas for the conversational text-based question-answering, the query rewrite approach performs best.