Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Choice-75: A Dataset on Decision Branching in Script Learning

Zhaoyi Joey Hou, Li Zhang, Chris Callison-Burch

TL;DR

Choice-75 is proposed, the first benchmark that challenges intelligent systems to make decisions given descriptive scenarios, containing 75 scripts and more than 600 scenarios, and preliminary results with current large language models (LLM) are presented.

Abstract

Script learning studies how stereotypical events unfold, enabling machines to reason about narratives with implicit information. Previous works mostly consider a script as a linear sequence of events while ignoring the potential branches that arise due to people's circumstantial choices. We hence propose Choice-75, the first benchmark that challenges intelligent systems to make decisions given descriptive scenarios, containing 75 scripts and more than 600 scenarios. We also present preliminary results with current large language models (LLM). Although they demonstrate overall decent performance, there is still notable headroom in hard scenarios.

Choice-75: A Dataset on Decision Branching in Script Learning

TL;DR

Choice-75 is proposed, the first benchmark that challenges intelligent systems to make decisions given descriptive scenarios, containing 75 scripts and more than 600 scenarios, and preliminary results with current large language models (LLM) are presented.

Abstract

Script learning studies how stereotypical events unfold, enabling machines to reason about narratives with implicit information. Previous works mostly consider a script as a linear sequence of events while ignoring the potential branches that arise due to people's circumstantial choices. We hence propose Choice-75, the first benchmark that challenges intelligent systems to make decisions given descriptive scenarios, containing 75 scripts and more than 600 scenarios. We also present preliminary results with current large language models (LLM). Although they demonstrate overall decent performance, there is still notable headroom in hard scenarios.
Paper Structure (18 sections, 3 figures, 4 tables)

This paper contains 18 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Dataset
  3. Overview
  4. Difficulty Level
  5. Manual Scenario Annotation
  6. Human-in-the-Loop Generation
  7. Method and Experiments
  8. Results and Analysis
  9. Difficulty Levels
  10. Human Performance
  11. Case Studies
  12. Related Work
  13. Conclusion
  14. Inter-Annotator Agreement
  15. Human-in-the-loop Data Generation Prompting Details
  16. ...and 3 more sections

Figures (3)

  • Figure 1: An example of Choice-75. Each goal-option pair has multiple scenarios.
  • Figure 2: Hard scenario generation (verb phrase)
  • Figure 3: Hard scenario generation (user profile)