Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can General-Purpose Large Language Models Generalize to English-Thai Machine Translation ?

Jirat Chiaranaipanich, Naiyarat Hanmatheekuna, Jitkapat Sawatphol, Krittamate Tiankanon, Jiramet Kinchagawat, Amrest Chinkamol, Parinthapat Pengpun, Piyalitt Ittichaiwong, Peerat Limkonchotiwat

TL;DR

It is revealed that under more strict computational constraints, such as 4-bit quantization, LLMs fail to translate effectively, and specialized models, with comparable or lower computational requirements, consistently outperform LLMs.

Abstract

Large language models (LLMs) perform well on common tasks but struggle with generalization in low-resource and low-computation settings. We examine this limitation by testing various LLMs and specialized translation models on English-Thai machine translation and code-switching datasets. Our findings reveal that under more strict computational constraints, such as 4-bit quantization, LLMs fail to translate effectively. In contrast, specialized models, with comparable or lower computational requirements, consistently outperform LLMs. This underscores the importance of specialized models for maintaining performance under resource constraints.

Can General-Purpose Large Language Models Generalize to English-Thai Machine Translation ?

TL;DR

It is revealed that under more strict computational constraints, such as 4-bit quantization, LLMs fail to translate effectively, and specialized models, with comparable or lower computational requirements, consistently outperform LLMs.

Abstract

Large language models (LLMs) perform well on common tasks but struggle with generalization in low-resource and low-computation settings. We examine this limitation by testing various LLMs and specialized translation models on English-Thai machine translation and code-switching datasets. Our findings reveal that under more strict computational constraints, such as 4-bit quantization, LLMs fail to translate effectively. In contrast, specialized models, with comparable or lower computational requirements, consistently outperform LLMs. This underscores the importance of specialized models for maintaining performance under resource constraints.

Paper Structure

This paper contains 8 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Results
  4. Analysis
  5. Conclusion
  6. Appendix
  7. Finetuning Prompts for Llama
  8. LLM Judge Prompts

Figures (1)

  • Figure 1: Llama-3 and NLLB failure analysis. Note that the legend is shared between Figures \ref{['fig:csllmjudgecomparison']} and \ref{['fig:scbllmjudgecomparison']}.