Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia

Fajri Koto

TL;DR

IndoCareer introduces the first large-scale, Indonesian professional-exam benchmark comprising 8,834 MCQs across 22 professions in six domains to evaluate LLM readiness for real-world tasks. The study benchmarks 27 LLMs using zero-shot Indonesian prompts, revealing that top models like GPT-4o and LLaMA-3.1 substantially outperform Indonesian-centric adaptations, while healthcare and finance domains remain particularly challenging. A key finding is that option-order shuffling destabilizes performance in insurance and finance, and that questions requiring local context or numerical reasoning disproportionately challenge models, indicating gaps in domain adaptation and numerical cognition. The work underscores the necessity of targeted fine-tuning and domain-aware evaluation to bridge the gap between current LLM capabilities and practical professional requirements in Indonesia, with broad implications for vocational NLP benchmarks in underrepresented languages.

Abstract

While knowledge evaluation in large language models has predominantly focused on academic subjects like math and physics, these assessments often fail to capture the practical demands of real-world professions. In this paper, we introduce IndoCareer, a dataset comprising 8,834 multiple-choice questions designed to evaluate performance in vocational and professional certification exams across various fields. With a focus on Indonesia, IndoCareer provides rich local contexts, spanning six key sectors: (1) healthcare, (2) insurance and finance, (3) creative and design, (4) tourism and hospitality, (5) education and training, and (6) law. Our comprehensive evaluation of 27 large language models shows that these models struggle particularly in fields with strong local contexts, such as insurance and finance. Additionally, while using the entire dataset, shuffling answer options generally maintains consistent evaluation results across models, but it introduces instability specifically in the insurance and finance sectors.

Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia

TL;DR

IndoCareer introduces the first large-scale, Indonesian professional-exam benchmark comprising 8,834 MCQs across 22 professions in six domains to evaluate LLM readiness for real-world tasks. The study benchmarks 27 LLMs using zero-shot Indonesian prompts, revealing that top models like GPT-4o and LLaMA-3.1 substantially outperform Indonesian-centric adaptations, while healthcare and finance domains remain particularly challenging. A key finding is that option-order shuffling destabilizes performance in insurance and finance, and that questions requiring local context or numerical reasoning disproportionately challenge models, indicating gaps in domain adaptation and numerical cognition. The work underscores the necessity of targeted fine-tuning and domain-aware evaluation to bridge the gap between current LLM capabilities and practical professional requirements in Indonesia, with broad implications for vocational NLP benchmarks in underrepresented languages.

Abstract

While knowledge evaluation in large language models has predominantly focused on academic subjects like math and physics, these assessments often fail to capture the practical demands of real-world professions. In this paper, we introduce IndoCareer, a dataset comprising 8,834 multiple-choice questions designed to evaluate performance in vocational and professional certification exams across various fields. With a focus on Indonesia, IndoCareer provides rich local contexts, spanning six key sectors: (1) healthcare, (2) insurance and finance, (3) creative and design, (4) tourism and hospitality, (5) education and training, and (6) law. Our comprehensive evaluation of 27 large language models shows that these models struggle particularly in fields with strong local contexts, such as insurance and finance. Additionally, while using the entire dataset, shuffling answer options generally maintains consistent evaluation results across models, but it introduces instability specifically in the insurance and finance sectors.
Paper Structure (17 sections, 3 figures, 6 tables)

This paper contains 17 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Indonesian Language Models
  4. Benchmarks for Evaluating Language Models
  5. IndoCareer
  6. Data Construction
  7. Quality Control
  8. Data Statistics
  9. Experiments
  10. Results
  11. Analysis
  12. Shuffling the multiple-choice options leads to unstable results in insurance and finance.
  13. LLMs perform well in life insurance certification but struggle with finance-related certifications.
  14. Questions with local context and numerical analysis pose greater challenges.
  15. Conclusion
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Distribution of professions in IndoCareer.
  • Figure 2: Example of questions in IndoCareer. The English translation is only for illustrative purposes.
  • Figure 3: Top 5 and bottom 5 professions based on the model's accuracy disparity relative to the passing score.