An Open-Source Web-Based Tool for Evaluating Open-Source Large Language Models Leveraging Information Retrieval from Custom Documents
Godfrey I
TL;DR
This work addresses the gap in pragmatic understanding of open-source LLMs by introducing an open-source, web-based tool that jointly models user speech acts and document-based retrieval to guide responses. The approach combines multi-model support (five Ollama-backed LLMs), speech-act extraction, and knowledge-text prompting within an interactive interface, evaluated on two domain documents using a comprehensive set of metrics (ROUGE, QA-Ref/Cand, BERT-based scores, METEOR, Perplexity). Results indicate larger models benefit from including speech acts, achieving better alignment with knowledge documents, while smaller models exhibit higher perplexity and mixed performance, underscoring model-specific optimization needs and cost considerations. Overall, the resource demonstrates the potential of integrating communicative intent and retrieval-based context to enhance conversational AI and provides a foundation for scalable, context-aware HCI applications.
Abstract
In our work, we present the first-of-its-kind open-source web-based tool which is able to demonstrate the impacts of a user's speech act during discourse with conversational agents, which leverages open-source large language models. With this software resource, it is possible for researchers and experts to evaluate the performance of various dialogues, visualize the user's communicative intents, and utilise uploaded specific documents for the chat agent to use for its information retrieval to respond to the user query. The context gathered by these models is obtained from a set of linguistic features extracted, which forms the context embeddings of the models. Regardless of these models showing good context understanding based on these features, there still remains a gap in including deeper pragmatic features to improve the model's comprehension of the query, hence the efforts to develop this web resource, which is able to extract and then inject this overlooked feature in the encoder-decoder pipeline of the conversational agent. To demonstrate the effect and impact of the resource, we carried out an experiment which evaluated the system using 2 knowledge files for information retrieval, with two user queries each, across 5 open-source large language models using 10 standard metrics. Our results showed that larger open-source models, demonstrated an improved alignment when the user speech act was included with their query. The smaller models in contrast showed an increased perplexity and mixed performance, which explicitly indicated struggles in processing queries that explicitly included speech acts. The results from the analysis using the developed web resource highlight the potential of speech acts towards enhancing conversational depths while underscoring the need for model-specific optimizations to address increased computational costs and response times.