Neural Machine Translation for Malayalam Paraphrase Generation

TL;DR

This study investigates Malayalam paraphrase generation by leveraging four English-based paraphrasing pipelines and pre-trained NMT models, evaluated with both automated metrics and human judgments. Using the English GYAFC dataset as a seed, plus a crowd-labeled Malayalam paraphrase set, it compares translation-based and language-specific approaches, including MultiIndic Paraphrase Generation, synonym-based English paraphrasing, BART-large-CNN, and OPUS MT with beam search. A key finding is that automated metrics (BLEU, METEOR, cosine) often fail to align with human judgments in Malayalam, while translation-based pipelines can perform comparably to Malayalam-specific paraphrase methods; human evaluation reveals some simple English-to-Malayalam heuristics can yield high-quality paraphrases. The authors release a human-labeled Malayalam paraphrase dataset (800 pairs) to foster future research and emphasize the need for evaluation metrics tailored to Dravidian languages and agglutinative morphology.

Abstract

This study explores four methods of generating paraphrases in Malayalam, utilizing resources available for English paraphrasing and pre-trained Neural Machine Translation (NMT) models. We evaluate the resulting paraphrases using both automated metrics, such as BLEU, METEOR, and cosine similarity, as well as human annotation. Our findings suggest that automated evaluation measures may not be fully appropriate for Malayalam, as they do not consistently align with human judgment. This discrepancy underscores the need for more nuanced paraphrase evaluation approaches especially for highly agglutinative languages.

Figures (4)

• Figure 1: Result from Model 1
• Figure 2: Result from Model 2
• Figure 3: Result from Model 3
• Figure 4: Result from Model 4