Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese

Rifki Afina Putri, Faiz Ghifari Haznitrama, Dea Adhista, Alice Oh

TL;DR

This study investigates the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages and finds that GPT-4 Turbo can generate questions with adequate general knowledge in both languages, albeit not as culturally ‘deep’ as humans.

Abstract

Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models. However, it is unclear whether they can generate a good quality of question answering (QA) dataset that incorporates knowledge and cultural nuance embedded in a language, especially for low-resource languages. In this study, we investigate the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages. To do so, we create datasets for these languages using various methods involving both LLMs and human annotators, resulting in ~4.5K questions per language (~9K in total), making our dataset the largest of its kind. Our experiments show that automatic data adaptation from an existing English dataset is less effective for Sundanese. Interestingly, using the direct generation method on the target language, GPT-4 Turbo can generate questions with adequate general knowledge in both languages, albeit not as culturally 'deep' as humans. We also observe a higher occurrence of fluency errors in the Sundanese dataset, highlighting the discrepancy between medium- and lower-resource languages.

Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese

TL;DR

This study investigates the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages and finds that GPT-4 Turbo can generate questions with adequate general knowledge in both languages, albeit not as culturally ‘deep’ as humans.

Abstract

Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models. However, it is unclear whether they can generate a good quality of question answering (QA) dataset that incorporates knowledge and cultural nuance embedded in a language, especially for low-resource languages. In this study, we investigate the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages. To do so, we create datasets for these languages using various methods involving both LLMs and human annotators, resulting in ~4.5K questions per language (~9K in total), making our dataset the largest of its kind. Our experiments show that automatic data adaptation from an existing English dataset is less effective for Sundanese. Interestingly, using the direct generation method on the target language, GPT-4 Turbo can generate questions with adequate general knowledge in both languages, albeit not as culturally 'deep' as humans. We also observe a higher occurrence of fluency errors in the Sundanese dataset, highlighting the discrepancy between medium- and lower-resource languages.
Paper Structure (69 sections, 8 figures, 13 tables)

This paper contains 69 sections, 8 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Commonsense Dataset
  4. Cultural Evaluation Dataset
  5. Background
  6. Commonsense QA: Definition and Scope
  7. Languages in Indonesia
  8. Data Generation Methods
  9. Automatic Data Adaptation
  10. Data Selection
  11. QA Pairs Adaptation
  12. Data Filtering
  13. Manual Data Generation
  14. QA Pairs Creation
  15. Answering Question
  16. ...and 54 more sections

Figures (8)

  • Figure 1: Our dataset generation methods. The examples of LLM_Adapt, Human_Gen, and LLM_Gen datasets are shown in English for clarity. The original versions of these datasets are in Indonesian and Sundanese.
  • Figure 2: Top-10 adapted question concepts taken from train, validation, and test set of LLM_Adapt data.
  • Figure 3: LLMs' performance on our combined test set.
  • Figure 4: LLMs' performance on LLM_Gen vs. Human_Gen in Indonesian and Sundanese. We combined data points from both languages for visualization, with lower quartiles typically representing Sundanese data.
  • Figure 5: LLMs performance by question category in LLM_Gen and Human_Gen for Indonesian and Sundanese.
  • ...and 3 more figures