Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Artificial Humans

Birger Moell

TL;DR

The paper benchmarks a Character.ai-based chatbot acting as a clinical psychologist to evaluate human-likeness and engagement and to assess progress toward AGI in conversational AI. Using a 27-participant survey across professionals and the public, the study finds the system can be engaging and reasonably human-like for some users but exhibits clear empathy limitations and generic responses. These findings highlight both the progress and the gaps in current AI conversation capabilities, emphasizing the need for improved prompting, empathy modeling, and interaction design before approaching true AI-therapist capabilities. The work informs future research directions in clinical-HCI applications and prompts further exploration of how to bridge the gap to AGI in dialogue systems.

Abstract

This study investigates the development and assessment of an artificial human designed as a conversational AI chatbot, focusing on its role as a clinical psychologist. The project involved creating a specialized chatbot using the Character.ai platform. The chatbot was designed to engage users in psychological discussions, providing advice and support with a human-like touch. The study involved participants (N=27) from diverse backgrounds, including psychologists, AI researchers, and the general public, who interacted with the chatbot and provided feedback on its human-likeness, empathy, and engagement levels. Results indicate that while many users found the chatbot engaging and somewhat human-like, limitations were noted in areas such as empathy and nuanced understanding. The findings suggest that although conversational AI has made strides, it remains far from achieving the true human-like interaction necessary for Artificial General Intelligence (AGI). The study highlights the challenges and potential of AI in human-computer interactions, suggesting directions for future research and development to bridge the gap between current capabilities and AGI. The project was completed in November of 2022 before the release of chatGPT.

Artificial Humans

TL;DR

The paper benchmarks a Character.ai-based chatbot acting as a clinical psychologist to evaluate human-likeness and engagement and to assess progress toward AGI in conversational AI. Using a 27-participant survey across professionals and the public, the study finds the system can be engaging and reasonably human-like for some users but exhibits clear empathy limitations and generic responses. These findings highlight both the progress and the gaps in current AI conversation capabilities, emphasizing the need for improved prompting, empathy modeling, and interaction design before approaching true AI-therapist capabilities. The work informs future research directions in clinical-HCI applications and prompts further exploration of how to bridge the gap to AGI in dialogue systems.

Abstract

This study investigates the development and assessment of an artificial human designed as a conversational AI chatbot, focusing on its role as a clinical psychologist. The project involved creating a specialized chatbot using the Character.ai platform. The chatbot was designed to engage users in psychological discussions, providing advice and support with a human-like touch. The study involved participants (N=27) from diverse backgrounds, including psychologists, AI researchers, and the general public, who interacted with the chatbot and provided feedback on its human-likeness, empathy, and engagement levels. Results indicate that while many users found the chatbot engaging and somewhat human-like, limitations were noted in areas such as empathy and nuanced understanding. The findings suggest that although conversational AI has made strides, it remains far from achieving the true human-like interaction necessary for Artificial General Intelligence (AGI). The study highlights the challenges and potential of AI in human-computer interactions, suggesting directions for future research and development to bridge the gap between current capabilities and AGI. The project was completed in November of 2022 before the release of chatGPT.

Paper Structure

This paper contains 8 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Methodology
  3. Technical Details
  4. Results
  5. Discussion
  6. Conclusion
  7. What was your reflection after talking to the chatbot?
  8. Additional Participant Feedback

Figures (5)

  • Figure 1: Familiarity with psychology
  • Figure 2: Familiarity with AI
  • Figure 3: Example conversation with the chatbot
  • Figure 4: How human-like was the conversation?
  • Figure 5: How engaging was the conversation?