Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Questionnaires for Everyone: Streamlining Cross-Cultural Questionnaire Adaptation with GPT-Based Translation Quality Evaluation

Otso Haavisto, Robin Welsch

TL;DR

This paper presents a prototype tool that integrates forward-backward translation with GPT-4 translation-quality evaluations to streamline cross-cultural questionnaire adaptation. Two online studies (English→German and English→Portuguese) show that GPT-4-based evaluations can help users attain translation quality comparable to conventional methods, reducing resource demands. The work demonstrates the viability of GEMBA-DA and SSA as actionable feedback mechanisms and discusses usability, limitations, and design considerations for AI-assisted translation tooling. Overall, it argues for AI-powered enhancements to make questionnaire-based research more equitable across languages and contexts, while outlining paths for broader validation and longer instruments.

Abstract

Adapting questionnaires to new languages is a resource-intensive process often requiring the hiring of multiple independent translators, which limits the ability of researchers to conduct cross-cultural research and effectively creates inequalities in research and society. This work presents a prototype tool that can expedite the questionnaire translation process. The tool incorporates forward-backward translation using DeepL alongside GPT-4-generated translation quality evaluations and improvement suggestions. We conducted two online studies in which participants translated questionnaires from English to either German (Study 1; n=10) or Portuguese (Study 2; n=20) using our prototype. To evaluate the quality of the translations created using the tool, evaluation scores between conventionally translated and tool-supported versions were compared. Our results indicate that integrating LLM-generated translation quality evaluations and suggestions for improvement can help users independently attain results similar to those provided by conventional, non-NLP-supported translation methods. This is the first step towards more equitable questionnaire-based research, powered by AI.

Questionnaires for Everyone: Streamlining Cross-Cultural Questionnaire Adaptation with GPT-Based Translation Quality Evaluation

TL;DR

This paper presents a prototype tool that integrates forward-backward translation with GPT-4 translation-quality evaluations to streamline cross-cultural questionnaire adaptation. Two online studies (English→German and English→Portuguese) show that GPT-4-based evaluations can help users attain translation quality comparable to conventional methods, reducing resource demands. The work demonstrates the viability of GEMBA-DA and SSA as actionable feedback mechanisms and discusses usability, limitations, and design considerations for AI-assisted translation tooling. Overall, it argues for AI-powered enhancements to make questionnaire-based research more equitable across languages and contexts, while outlining paths for broader validation and longer instruments.

Abstract

Adapting questionnaires to new languages is a resource-intensive process often requiring the hiring of multiple independent translators, which limits the ability of researchers to conduct cross-cultural research and effectively creates inequalities in research and society. This work presents a prototype tool that can expedite the questionnaire translation process. The tool incorporates forward-backward translation using DeepL alongside GPT-4-generated translation quality evaluations and improvement suggestions. We conducted two online studies in which participants translated questionnaires from English to either German (Study 1; n=10) or Portuguese (Study 2; n=20) using our prototype. To evaluate the quality of the translations created using the tool, evaluation scores between conventionally translated and tool-supported versions were compared. Our results indicate that integrating LLM-generated translation quality evaluations and suggestions for improvement can help users independently attain results similar to those provided by conventional, non-NLP-supported translation methods. This is the first step towards more equitable questionnaire-based research, powered by AI.
Paper Structure (26 sections, 12 figures, 1 table)

This paper contains 26 sections, 12 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Prototype
  4. Task Analysis
  5. Prototype Construction
  6. Study 1: Prototype-assisted Translation of the ATI Scale from English to German
  7. Method
  8. Participants
  9. Analysis
  10. Results
  11. Evaluation Scores
  12. Participant Experience
  13. Study 2: Prototype-assisted Translation of the BFI-10 Scale from English to Portuguese
  14. Method
  15. Participants
  16. ...and 11 more sections

Figures (12)

  • Figure 1: The view for editing and evaluating translations in the final prototype application.
  • Figure 2: Brislin's brislin_back-translation_1970 backtranslation approach
  • Figure 3: Brislin's brislin_back-translation_1970 committee approach
  • Figure 4: Finnish ATI translation process heilala_finnish_2023
  • Figure 5: Finnish CLEFT-Q translation process westerlund_finnish_2023
  • ...and 7 more figures