Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VicSim: Enhancing Victim Simulation with Emotional and Linguistic Fidelity

Yerong Li, Yiren Liu, Yun Huang

TL;DR

VicSim introduces a victim simulator for scenario-based training that targets informational faithfulness, emotional dynamics, and linguistic fidelity. It combines a GAN-based adversarial framework with key-information prompting to reduce hallucinations and align generated text with scenario specifics. Through automatic metrics and human judgments, VicSim outperforms GPT-4 in human likeness and informational fidelity, though challenges remain when scenario details are sparse. The work demonstrates practical potential for dispatcher training and highlights avenues for multi-modal, curriculum-integrated future research.

Abstract

Scenario-based training has been widely adopted in many public service sectors. Recent advancements in Large Language Models (LLMs) have shown promise in simulating diverse personas to create these training scenarios. However, little is known about how LLMs can be developed to simulate victims for scenario-based training purposes. In this paper, we introduce VicSim (victim simulator), a novel model that addresses three key dimensions of user simulation: informational faithfulness, emotional dynamics, and language style (e.g., grammar usage). We pioneer the integration of scenario-based victim modeling with GAN-based training workflow and key-information-based prompting, aiming to enhance the realism of simulated victims. Our adversarial training approach teaches the discriminator to recognize grammar and emotional cues as reliable indicators of synthetic content. According to evaluations by human raters, the VicSim model outperforms GPT-4 in terms of human-likeness.

VicSim: Enhancing Victim Simulation with Emotional and Linguistic Fidelity

TL;DR

VicSim introduces a victim simulator for scenario-based training that targets informational faithfulness, emotional dynamics, and linguistic fidelity. It combines a GAN-based adversarial framework with key-information prompting to reduce hallucinations and align generated text with scenario specifics. Through automatic metrics and human judgments, VicSim outperforms GPT-4 in human likeness and informational fidelity, though challenges remain when scenario details are sparse. The work demonstrates practical potential for dispatcher training and highlights avenues for multi-modal, curriculum-integrated future research.

Abstract

Scenario-based training has been widely adopted in many public service sectors. Recent advancements in Large Language Models (LLMs) have shown promise in simulating diverse personas to create these training scenarios. However, little is known about how LLMs can be developed to simulate victims for scenario-based training purposes. In this paper, we introduce VicSim (victim simulator), a novel model that addresses three key dimensions of user simulation: informational faithfulness, emotional dynamics, and language style (e.g., grammar usage). We pioneer the integration of scenario-based victim modeling with GAN-based training workflow and key-information-based prompting, aiming to enhance the realism of simulated victims. Our adversarial training approach teaches the discriminator to recognize grammar and emotional cues as reliable indicators of synthetic content. According to evaluations by human raters, the VicSim model outperforms GPT-4 in terms of human-likeness.
Paper Structure (31 sections, 2 equations, 9 figures, 3 tables)

This paper contains 31 sections, 2 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. User Simulation
  4. AI-based Solutions for Simulating Scenario-based Interactions
  5. Method
  6. VicSim: Simulating Victims' Text Reporting
  7. Dataset
  8. Data Collection
  9. Data Ethics
  10. Evaluation
  11. Assessment of Informational faithfulness
  12. Assessment of User Emotions
  13. Assessment of Grammar styles
  14. Human Evaluation
  15. Findings
  16. ...and 16 more sections

Figures (9)

  • Figure 1: GAN training workflow : we used Flan-T5 based discriminator and Llama-2 chat-based simulated user as the generator
  • Figure 2: Illustration of the prompt construction process for simulated user dialogue generation: we concatenate the system guidance, user's scenario, along with dialogue history, enabling comprehensive prompts for dialogue generation by the LLM
  • Figure 3: Senario-based prompt argumented by the key information extracted from the CoreNLP toolkit
  • Figure 4: Users' utterances at different conversation stages on the evaluation set
  • Figure 5: Keyword-based assessment on hallucination, here [John Smith], [Cortright] are simulated key information we filled in for data processing.
  • ...and 4 more figures