Nationality and Region Prediction from Names: A Comparative Study of Neural Models and Large Language Models
Keito Inoshita
TL;DR
The paper systematically compares neural models and prompting-based large language models for predicting nationality and region from names, across fine-, mid-, and coarse-grained targets. It demonstrates that LLMs generally exceed neural models, with the performance gap narrowing as granularity becomes coarser, and reveals distinct error patterns where LLMs tend toward near-miss regional predictions while neural models favor high-frequency nationalities. Frequency robustness analysis shows simple pattern-based methods are most stable across low-frequency classes, whereas pre-trained and LLM-based approaches exhibit biases tied to training data distributions. The findings offer practical guidance on model selection by task granularity and advocate evaluating error quality in addition to accuracy, with implications for designing hybrid or ensemble systems that leverage complementary strengths of neural models and LLMs.
Abstract
Predicting nationality from personal names has practical value in marketing, demographic research, and genealogical studies. Conventional neural models learn statistical correspondences between names and nationalities from task-specific training data, posing challenges in generalizing to low-frequency nationalities and distinguishing similar nationalities within the same region. Large language models (LLMs) have the potential to address these challenges by leveraging world knowledge acquired during pre-training. In this study, we comprehensively compare neural models and LLMs on nationality prediction, evaluating six neural models and six LLM prompting strategies across three granularity levels (nationality, region, and continent), with frequency-based stratified analysis and error analysis. Results show that LLMs outperform neural models at all granularity levels, with the gap narrowing as granularity becomes coarser. Simple machine learning methods exhibit the highest frequency robustness, while pre-trained models and LLMs show degradation for low-frequency nationalities. Error analysis reveals that LLMs tend to make ``near-miss'' errors, predicting the correct region even when nationality is incorrect, whereas neural models exhibit more cross-regional errors and bias toward high-frequency classes. These findings indicate that LLM superiority stems from world knowledge, model selection should consider required granularity, and evaluation should account for error quality beyond accuracy.