StorySparkQA: Expert-Annotated QA Pairs with Real-World Knowledge for Children's Story-Based Learning

TL;DR

This work designs an annotation framework, empowered by existing knowledge graph to capture experts’ annotations and thinking process, and uses this framework to construct StorySparkQA dataset, which comprises 5, 868 expert-annotated QA pairs with real-world knowledge.

Abstract

Interactive story reading is a common parent-child activity, where parents expect to teach both language skills and real-world knowledge beyond the story. While increasing storytelling and reading systems have been developed for this activity, they often fail to infuse real-world knowledge into the conversation. This limitation can be attributed to the existing question-answering (QA) datasets used for children's education, upon which the systems are built, failing to capture the nuances of how education experts think when conducting interactive story reading activities. To bridge this gap, we design an annotation framework, empowered by existing knowledge graph to capture experts' annotations and thinking process, and leverage this framework to construct StorySparkQA dataset, which comprises 5,868 expert-annotated QA pairs with real-world knowledge. We conduct automated and human expert evaluations across various QA pair generation settings to demonstrate that our StorySparkQA can effectively support models in generating QA pairs that target real-world knowledge beyond story content. StorySparkQA is available at https://huggingface.co/datasets/NEU-HAI/StorySparkQA.

Figures (10)

• Figure 1: An example of StorySparkQA dataset. In each story section, educational experts select a concept word, link it to a desired external real-world knowledge, and write an appropriate QA pair. Additional data examples of StorySparkQA are presented in Appendix \ref{['app: sample']}.
• Figure 2: Workflow of the experts’ annotation process. Experts need to select a concept first, then match it with the most suitable knowledge, and finally create a QA pair based on the selected knowledge.
• Figure 3: The user interface to facilitate our annotation task. The words highlighted in grey are candidate concepts. The blue block shows the Wiktionary explanation, and the yellow block lists our recommended triples.
• Figure 4: Distribution of real-world knowledge relations annotated by experts in the StorySparkQA dataset
• Figure 5: Annotation process 1: browse a displayed section, with candidate words highlighted in grey.