Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Crowd-sourcing NLG Data: Pictures Elicit Better Data

Jekaterina Novikova, Oliver Lemon, Verena Rieser

TL;DR

This study investigates crowdsourcing high-quality NLG training data by comparing textual/logical MRs with pictorial MRs. It introduces automatic pre-validation and human evaluation to ensure data quality and demonstrates that pictorial MRs elicit more informative, natural, and better-phrased utterances, especially as MR complexity increases. The authors collect 1410 utterances across two MR modalities and show that pictorial stimuli reduce lexical priming and promote varied language, with a robust evaluation framework validating data quality. The work provides a practical framework for rapid NLG data creation and offers a dataset for future corpus-based NLG methods, including imitation-learning approaches.

Abstract

Recent advances in corpus-based Natural Language Generation (NLG) hold the promise of being easily portable across domains, but require costly training data, consisting of meaning representations (MRs) paired with Natural Language (NL) utterances. In this work, we propose a novel framework for crowdsourcing high quality NLG training data, using automatic quality control measures and evaluating different MRs with which to elicit data. We show that pictorial MRs result in better NL data being collected than logic-based MRs: utterances elicited by pictorial MRs are judged as significantly more natural, more informative, and better phrased, with a significant increase in average quality ratings (around 0.5 points on a 6-point scale), compared to using the logical MRs. As the MR becomes more complex, the benefits of pictorial stimuli increase. The collected data will be released as part of this submission.

Crowd-sourcing NLG Data: Pictures Elicit Better Data

TL;DR

This study investigates crowdsourcing high-quality NLG training data by comparing textual/logical MRs with pictorial MRs. It introduces automatic pre-validation and human evaluation to ensure data quality and demonstrates that pictorial MRs elicit more informative, natural, and better-phrased utterances, especially as MR complexity increases. The authors collect 1410 utterances across two MR modalities and show that pictorial stimuli reduce lexical priming and promote varied language, with a robust evaluation framework validating data quality. The work provides a practical framework for rapid NLG data creation and offers a dataset for future corpus-based NLG methods, including imitation-learning approaches.

Abstract

Recent advances in corpus-based Natural Language Generation (NLG) hold the promise of being easily portable across domains, but require costly training data, consisting of meaning representations (MRs) paired with Natural Language (NL) utterances. In this work, we propose a novel framework for crowdsourcing high quality NLG training data, using automatic quality control measures and evaluating different MRs with which to elicit data. We show that pictorial MRs result in better NL data being collected than logic-based MRs: utterances elicited by pictorial MRs are judged as significantly more natural, more informative, and better phrased, with a significant increase in average quality ratings (around 0.5 points on a 6-point scale), compared to using the logical MRs. As the MR becomes more complex, the benefits of pictorial stimuli increase. The collected data will be released as part of this submission.

Paper Structure

This paper contains 19 sections, 1 equation, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Experimental setup
  4. Data collection: pictures and text
  5. Validation procedures
  6. Automatic Pre-validation
  7. Human evaluation of collected data
  8. Results: Collected data
  9. Time taken to collect data
  10. Average length of utterance (characters)
  11. Average number of sentences per utterance
  12. Semantic similarity
  13. Results: human evaluation of the collected data
  14. Self-evaluation vs. Crowd evaluation
  15. Informativeness
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Example of an MR-NL pair.
  • Figure 2: Examples of pictorial MRs. Left: for a family-friendly, Sushi/Japanese restaurant, cheap, neither near the centre of town nor near the river. Right: for a restaurant by the river, serving pasta/Italian food, highly rated and expensive, not child-friendly, located near Cafe Adriatic.