Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

JobResQA: A Benchmark for LLM Machine Reading Comprehension on Multilingual Résumés and JDs

Casimiro Pio Carrino, Paula Estrella, Rabih Zbib, Carlos Escolano, José A. R. Fonollosa

TL;DR

JobResQA addresses the need for robust, multilingual evaluation of LLM-based machine reading comprehension in HR workflows by providing a synthetic, bias-controllable dataset of résumé-JD pairs spanning five languages. It introduces a data generation pipeline that de-identifies real-world data, synthesizes realistic yet anonymized content, and a TEaR-based human-in-the-loop translation process to produce high-quality parallel data. Baseline experiments across open-weight LLM families show that although English and Spanish performance is relatively strong, cross-language capabilities lag behind, highlighting gaps in multilingual MRC for HR. The benchmark is publicly available and aims to drive fair, reliable HR AI systems through reproducible evaluation.

Abstract

We introduce JobResQA, a multilingual Question Answering benchmark for evaluating Machine Reading Comprehension (MRC) capabilities of LLMs on HR-specific tasks involving résumés and job descriptions. The dataset comprises 581 QA pairs across 105 synthetic résumé-job description pairs in five languages (English, Spanish, Italian, German, and Chinese), with questions spanning three complexity levels from basic factual extraction to complex cross-document reasoning. We propose a data generation pipeline derived from real-world sources through de-identification and data synthesis to ensure both realism and privacy, while controlled demographic and professional attributes (implemented via placeholders) enable systematic bias and fairness studies. We also present a cost-effective, human-in-the-loop translation pipeline based on the TEaR methodology, incorporating MQM error annotations and selective post-editing to ensure an high-quality multi-way parallel benchmark. We provide a baseline evaluations across multiple open-weight LLM families using an LLM-as-judge approach revealing higher performances on English and Spanish but substantial degradation for other languages, highlighting critical gaps in multilingual MRC capabilities for HR applications. JobResQA provides a reproducible benchmark for advancing fair and reliable LLM-based HR systems. The benchmark is publicly available at: https://github.com/Avature/jobresqa-benchmark

JobResQA: A Benchmark for LLM Machine Reading Comprehension on Multilingual Résumés and JDs

TL;DR

JobResQA addresses the need for robust, multilingual evaluation of LLM-based machine reading comprehension in HR workflows by providing a synthetic, bias-controllable dataset of résumé-JD pairs spanning five languages. It introduces a data generation pipeline that de-identifies real-world data, synthesizes realistic yet anonymized content, and a TEaR-based human-in-the-loop translation process to produce high-quality parallel data. Baseline experiments across open-weight LLM families show that although English and Spanish performance is relatively strong, cross-language capabilities lag behind, highlighting gaps in multilingual MRC for HR. The benchmark is publicly available and aims to drive fair, reliable HR AI systems through reproducible evaluation.

Abstract

We introduce JobResQA, a multilingual Question Answering benchmark for evaluating Machine Reading Comprehension (MRC) capabilities of LLMs on HR-specific tasks involving résumés and job descriptions. The dataset comprises 581 QA pairs across 105 synthetic résumé-job description pairs in five languages (English, Spanish, Italian, German, and Chinese), with questions spanning three complexity levels from basic factual extraction to complex cross-document reasoning. We propose a data generation pipeline derived from real-world sources through de-identification and data synthesis to ensure both realism and privacy, while controlled demographic and professional attributes (implemented via placeholders) enable systematic bias and fairness studies. We also present a cost-effective, human-in-the-loop translation pipeline based on the TEaR methodology, incorporating MQM error annotations and selective post-editing to ensure an high-quality multi-way parallel benchmark. We provide a baseline evaluations across multiple open-weight LLM families using an LLM-as-judge approach revealing higher performances on English and Spanish but substantial degradation for other languages, highlighting critical gaps in multilingual MRC capabilities for HR applications. JobResQA provides a reproducible benchmark for advancing fair and reliable LLM-based HR systems. The benchmark is publicly available at: https://github.com/Avature/jobresqa-benchmark
Paper Structure (37 sections, 3 figures, 6 tables)

This paper contains 37 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. The JobResQA Dataset
  4. Main Characteristics
  5. Statistics and Data Fields
  6. Accessibility and Reproducibility
  7. Résumés and Job Synthesis
  8. Data Collection and Job Matching
  9. De-identification
  10. Synthetic Generation
  11. Manual Review and Selective Post-Editing
  12. Question-Answering Annotations
  13. Human-in-the-Loop Machine Translation Pipeline
  14. TEaR with Human-in-the-Loop
  15. Zero-Shot Translation.
  16. ...and 22 more sections

Figures (3)

  • Figure 1: Industry distribution of the final selection of matched 105 résumé-JD pairs, with 24 industries and 194 unique job titles.
  • Figure 2: Data synthesis pipeline for JobResQA: collection and matching of résumé-JD pairs, de-identification, LLM-based synthesis, manual review and post-processing.
  • Figure 3: Human-in-the-loop TEaR translation pipeline for JobResQA: zero-shot translation, MQM error annotation and corrections, few-shot estimation, few-shot refinement and selective post-editing.