ChaI-TeA: A Benchmark for Evaluating Autocompletion of Interactions with LLM-based Chatbots

TL;DR

ChaI-TeA tackles autocompletion for user turns in LLM-based chatbots by formalizing a sequential autocomplete task, assembling English conversation datasets (Open Assistant OASST and ShareGPT), and introducing offline metrics including saved@k and latency. The study benchmarks nine LM-based autocomplete systems, revealing that while models can generate plausible completions, ranking quality and latency remain bottlenecks; end-of-word completions, leveraging distant history, and allowing varying completion lengths improve user experience, but perplexity-based ranking is insufficient. Fine-tuning via methods such as LoRA yields additional gains, and the framework highlights practical guidance for latency-aware deployment and future research directions. Overall, ChaI-TeA provides a foundation for evaluating and improving chat interaction autocompletion in real-world LLM chatbots, with broad implications for reducing user typing effort and cognitive load.

Abstract

The rise of LLMs has deflected a growing portion of human-computer interactions towards LLM-based chatbots. The remarkable abilities of these models allow users to interact using long, diverse natural language text covering a wide range of topics and styles. Phrasing these messages is a time and effort consuming task, calling for an autocomplete solution to assist users. We introduce the task of chatbot interaction autocomplete. We present ChaI-TeA: CHat InTEraction Autocomplete; An autcomplete evaluation framework for LLM-based chatbot interactions. The framework includes a formal definition of the task, coupled with suitable datasets and metrics. We use the framework to evaluate After formally defining the task along with suitable datasets and metrics, we test 9 models on the defined auto completion task, finding that while current off-the-shelf models perform fairly, there is still much room for improvement, mainly in ranking of the generated suggestions. We provide insights for practitioners working on this task and open new research directions for researchers in the field. We release our framework to serve as a foundation for future research.

Figures (8)

• Figure 1: The chatbot interaction autocompletion task. Given the conversation history and the current turn's prefix, task is to suggest suitable completions.
• Figure 2: saved@k on OASST for varying $k$ values.
• Figure 3: saved@$k$ comparison between solutions suggesting completions after words and characters.
• Figure 4: acceptance rate comparison between solutions suggesting completions after words and characters.
• Figure 5: saved@k and acc. rate@k on ShareGPT for varying $k$ values.