Empirical Study of Symmetrical Reasoning in Conversational Chatbots

TL;DR

The paper investigates whether conversational chatbots powered by large language models can understand predicate symmetry, a nontrivial linguistic property. It employs in-context learning and the SIS dataset to evaluate five chatbots against human judgments without any fine-tuning. Gemini Advanced achieves the strongest alignment with human symmetry judgments (approximately 0.85 correlation) and shows stable performance across seven trials, while other chatbots exhibit more modest correlations. The findings illustrate both the potential of LLMs to mirror complex cognitive language aspects and the current limitations stemming from stochastic responses and context sensitivity.

Abstract

This work explores the capability of conversational chatbots powered by large language models (LLMs), to understand and characterize predicate symmetry, a cognitive linguistic function traditionally believed to be an inherent human trait. Leveraging in-context learning (ICL), a paradigm shift enabling chatbots to learn new tasks from prompts without re-training, we assess the symmetrical reasoning of five chatbots: ChatGPT 4, Huggingface chat AI, Microsoft's Copilot AI, LLaMA through Perplexity, and Gemini Advanced. Using the Symmetry Inference Sentence (SIS) dataset by Tanchip et al. (2020), we compare chatbot responses against human evaluations to gauge their understanding of predicate symmetry. Experiment results reveal varied performance among chatbots, with some approaching human-like reasoning capabilities. Gemini, for example, reaches a correlation of 0.85 with human scores, while providing a sounding justification for each symmetry evaluation. This study underscores the potential and limitations of LLMs in mirroring complex cognitive processes as symmetrical reasoning.

Figures (2)

• Figure 1: Correlation of the seven Gemini trials, using the same 400 prompts.
• Figure 2: Correlation of the seven Gemini trials with the averaged human scores.