Eir: Thai Medical Large Language Models

TL;DR

Eir-8B is an 8-billion-parameter Thai medical LLM built on LLaMA 3.1 Instruct-8B and enhanced with LoRA fine-tuning to specialize in Thai medical contexts. The authors curate a 100k-page Thai/English pretraining corpus with ICD-10 knowledge and extensive synthetic data, and they construct 266k fine-tuning samples alongside a 18-domain EHR task suite to bridge clinical language and hospital workflows. Through advanced prompting (few-shot, chain-of-thought, ensembling) and model merging via SLERP, Eir-8B achieves competitive to superior performance on MedQA, MedMCQA, PubMedQA, MMLU medical subsets, ThaiExam, and M3Exam benchmarks, surpassing open Thai LLMs by over 10% and GPT-4o by over 11% on clinically oriented tasks. The model is deployed in hospital networks with strong privacy protections, and while safety cautions remain, the work establishes a practical pathway for Thai healthcare AI and highlights the need for careful evaluation before real-world clinical deployment.

Abstract

We present Eir-8B, a large language model with 8 billion parameters, specifically designed to enhance the accuracy of handling medical tasks in the Thai language. This model focuses on providing clear and easy-to-understand answers for both healthcare professionals and patients, thereby improving the efficiency of diagnosis and treatment processes. Human evaluation was conducted to ensure that the model adheres to care standards and provides unbiased answers. To prioritize data security, the model is deployed within the hospital's internal network, ensuring both high security and faster processing speeds. The internal API connection is secured with encryption and strict authentication measures to prevent data leaks and unauthorized access. We evaluated several open-source large language models with 8 billion parameters on four medical benchmarks: MedQA, MedMCQA, PubMedQA, and the medical subset of MMLU. The best-performing baselines were used to develop Eir-8B. Our evaluation employed multiple questioning strategies, including zero-shot, few-shot, chain-of-thought reasoning, and ensemble/self-consistency voting methods. Our model outperformed commercially available Thai-language large language models by more than 10%. In addition, we developed enhanced model testing tailored for clinical use in Thai across 18 clinical tasks, where our model exceeded GPT-4o performance by more than 11%.

Figures (11)

• Figure 1: An overview of the workflow for synthetic data generation using ChatGPT.
• Figure 2: Overview Medical Prompt Workflow for Document Processing and Querying Using Embedding Model
• Figure 3: Distribution of medical dataset pairs with quality scores, illustrating that the majority received high scores. Data points with scores below 7 (left of the red line) were excluded from training.
• Figure 4: Distribution of Thai exam dataset pairs with quality scores, illustrating that the majority received high scores. Data points with scores below 7 (left of the red line) were excluded from training.
• Figure 5: Medical Question Domain Specifics