Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Reliable Common-Sense Reasoning Socialbot Built Using LLMs and Goal-Directed ASP

Yankai Zeng, Abhiramon Rajashekharan, Kinjal Basu, Huaduo Wang, Joaquín Arias, Gopal Gupta

TL;DR

The paper tackles the unreliability and topic drift of LLM-only chatbots by introducing AutoCompanion, a socialbot that translates natural language into predicates via an LLM and uses a goal-directed ASP backend (s(CASP)) to reason about conversation flow. It leverages the STAR framework to integrate LLM-based semantic parsing with ASP-driven control, enabling topic maintenance, coherent transitions, and relevant recommendations around movies, books, and people. Empirical evaluation demonstrates competitive latency and superior creativity and topic concentration compared with a strong LLM-only baseline, highlighting the advantages of a predicate-based knowledge base combined with explicit reasoning. The work suggests that combining LLMs with structured reasoning yields better reliability, scalability, and user engagement, with future plans for broader knowledge bases, field testing, and multimodal capabilities.

Abstract

The development of large language models (LLMs), such as GPT, has enabled the construction of several socialbots, like ChatGPT, that are receiving a lot of attention for their ability to simulate a human conversation. However, the conversation is not guided by a goal and is hard to control. In addition, because LLMs rely more on pattern recognition than deductive reasoning, they can give confusing answers and have difficulty integrating multiple topics into a cohesive response. These limitations often lead the LLM to deviate from the main topic to keep the conversation interesting. We propose AutoCompanion, a socialbot that uses an LLM model to translate natural language into predicates (and vice versa) and employs commonsense reasoning based on Answer Set Programming (ASP) to hold a social conversation with a human. In particular, we rely on s(CASP), a goal-directed implementation of ASP as the backend. This paper presents the framework design and how an LLM is used to parse user messages and generate a response from the s(CASP) engine output. To validate our proposal, we describe (real) conversations in which the chatbot's goal is to keep the user entertained by talking about movies and books, and s(CASP) ensures (i) correctness of answers, (ii) coherence (and precision) during the conversation, which it dynamically regulates to achieve its specific purpose, and (iii) no deviation from the main topic.

A Reliable Common-Sense Reasoning Socialbot Built Using LLMs and Goal-Directed ASP

TL;DR

The paper tackles the unreliability and topic drift of LLM-only chatbots by introducing AutoCompanion, a socialbot that translates natural language into predicates via an LLM and uses a goal-directed ASP backend (s(CASP)) to reason about conversation flow. It leverages the STAR framework to integrate LLM-based semantic parsing with ASP-driven control, enabling topic maintenance, coherent transitions, and relevant recommendations around movies, books, and people. Empirical evaluation demonstrates competitive latency and superior creativity and topic concentration compared with a strong LLM-only baseline, highlighting the advantages of a predicate-based knowledge base combined with explicit reasoning. The work suggests that combining LLMs with structured reasoning yields better reliability, scalability, and user engagement, with future plans for broader knowledge bases, field testing, and multimodal capabilities.

Abstract

The development of large language models (LLMs), such as GPT, has enabled the construction of several socialbots, like ChatGPT, that are receiving a lot of attention for their ability to simulate a human conversation. However, the conversation is not guided by a goal and is hard to control. In addition, because LLMs rely more on pattern recognition than deductive reasoning, they can give confusing answers and have difficulty integrating multiple topics into a cohesive response. These limitations often lead the LLM to deviate from the main topic to keep the conversation interesting. We propose AutoCompanion, a socialbot that uses an LLM model to translate natural language into predicates (and vice versa) and employs commonsense reasoning based on Answer Set Programming (ASP) to hold a social conversation with a human. In particular, we rely on s(CASP), a goal-directed implementation of ASP as the backend. This paper presents the framework design and how an LLM is used to parse user messages and generate a response from the s(CASP) engine output. To validate our proposal, we describe (real) conversations in which the chatbot's goal is to keep the user entertained by talking about movies and books, and s(CASP) ensures (i) correctness of answers, (ii) coherence (and precision) during the conversation, which it dynamically regulates to achieve its specific purpose, and (iii) no deviation from the main topic.
Paper Structure (16 sections, 1 figure, 3 tables)

This paper contains 16 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Answer Set Programming and s(CASP)
  4. State-of-the-art Chatbots and Their Limitations
  5. Socialbot Design
  6. LLM Parser and Ontology
  7. Topic Control
  8. Response Generation
  9. Implementation
  10. Data Collection
  11. ASP Coding
  12. Leveraging LLMs
  13. Conversation Samples
  14. Evaluation
  15. Discussion
  16. ...and 1 more sections

Figures (1)

  • Figure 1: AutoCompanion Architecture. The yellow-colored boxes are handled by GPT-4 in Python, and the green-colored by s(CASP). The two parts interact using Python subprocess calls.