Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Difficult for Whom? A Study of Japanese Lexical Complexity

Adam Nohejl, Akio Hayakawa, Yusuke Ide, Taro Watanabe

TL;DR

This work verifies that a recent Japanese LCP dataset is representative of its target population by partially replicating the annotation, and shows that native Chinese speakers perceive the complexity differently due to Sino-Japanese vocabulary.

Abstract

The tasks of lexical complexity prediction (LCP) and complex word identification (CWI) commonly presuppose that difficult to understand words are shared by the target population. Meanwhile, personalization methods have also been proposed to adapt models to individual needs. We verify that a recent Japanese LCP dataset is representative of its target population by partially replicating the annotation. By another reannotation we show that native Chinese speakers perceive the complexity differently due to Sino-Japanese vocabulary. To explore the possibilities of personalization, we compare competitive baselines trained on the group mean ratings and individual ratings in terms of performance for an individual. We show that the model trained on a group mean performs similarly to an individual model in the CWI task, while achieving good LCP performance for an individual is difficult. We also experiment with adapting a finetuned BERT model, which results only in marginal improvements across all settings.

Difficult for Whom? A Study of Japanese Lexical Complexity

TL;DR

This work verifies that a recent Japanese LCP dataset is representative of its target population by partially replicating the annotation, and shows that native Chinese speakers perceive the complexity differently due to Sino-Japanese vocabulary.

Abstract

The tasks of lexical complexity prediction (LCP) and complex word identification (CWI) commonly presuppose that difficult to understand words are shared by the target population. Meanwhile, personalization methods have also been proposed to adapt models to individual needs. We verify that a recent Japanese LCP dataset is representative of its target population by partially replicating the annotation. By another reannotation we show that native Chinese speakers perceive the complexity differently due to Sino-Japanese vocabulary. To explore the possibilities of personalization, we compare competitive baselines trained on the group mean ratings and individual ratings in terms of performance for an individual. We show that the model trained on a group mean performs similarly to an individual model in the CWI task, while achieving good LCP performance for an individual is difficult. We also experiment with adapting a finetuned BERT model, which results only in marginal improvements across all settings.

Paper Structure

This paper contains 15 sections, 4 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Analysis
  3. Target Population
  4. Correlation Analysis
  5. Experiments
  6. Frequency Baseline
  7. BERT-Based Model
  8. Conclusion
  9. Comparison of Word Origins and Parts of Speech in the Test and Trial Sets
  10. Handling of Words Missing in Data Sources
  11. Difference of Complexity Perception by Annotators' L1 and Word Origin
  12. Original Annotation and Chinese L1 Reannnotation
  13. Original Annotation and Replication
  14. Model Details
  15. Results of the BERT-based Model Variants

Figures (4)

  • Figure 1: Complexity histogram of the trial and test sets.
  • Figure 2: Inter-annotator agreement and mean pairwise correlation in the three annotator groups, the unions of their pairs, and union of all three. Light text denotes that union decreases agreement (correlation).
  • Figure 3: Mean complexity of target words in the the trial set of MultiLS-Japanese and in the Chinese L1 reannotation, plotted against log-frequency. Lines show linear fit with 95% confidence interval as a shaded area.
  • Figure 4: Mean complexity of target words in the the trial set of MultiLS-Japanese and in the replication. Lines show linear fit with 95% confidence interval as a shaded area.