Can formal argumentative reasoning enhance LLMs performances?
Federico Castagna, Isabel Sassoon, Simon Parsons
TL;DR
This work investigates whether integrating computational argumentation into LLM workflows via MQArgEng can boost reasoning performance without retraining. The pipeline uses an LLM to generate competing arguments, constructs an abstract argumentation framework, and employs the ASPARTIX solver to compute acceptable extensions that guide final outputs. Evaluated on MT-Bench with GPT-4 judging, MQArgEng achieves a moderate overall gain (average +$2.18\%$) across multiple topical categories, illustrating the feasibility and potential of non-intrusive reasoning augmentation. The study highlights advantages such as model-agnostic applicability and selective triggering, while acknowledging limitations in argument parsing and latency, pointing to promising avenues for future refinement and scaling.
Abstract
Recent years witnessed significant performance advancements in deep-learning-driven natural language models, with a strong focus on the development and release of Large Language Models (LLMs). These improvements resulted in better quality AI-generated output but rely on resource-expensive training and upgrading of models. Although different studies have proposed a range of techniques to enhance LLMs without retraining, none have considered computational argumentation as an option. This is a missed opportunity since computational argumentation is an intuitive mechanism that formally captures agents' interactions and the information conflict that may arise during such interplays, and so it seems well-suited for boosting the reasoning and conversational abilities of LLMs in a seamless manner. In this paper, we present a pipeline (MQArgEng) and preliminary study to evaluate the effect of introducing computational argumentation semantics on the performance of LLMs. Our experiment's goal was to provide a proof-of-concept and a feasibility analysis in order to foster (or deter) future research towards a fully-fledged argumentation engine plugin for LLMs. Exploratory results using the MT-Bench indicate that MQArgEng provides a moderate performance gain in most of the examined topical categories and, as such, show promise and warrant further research.