Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs

TL;DR

The paper addresses evaluating LLMs' understanding of Arabic poetry, a linguistically and culturally rich domain. It presents Fann or Flop, a dataset and benchmark spanning 12 eras, 14 genres, and multiple meters with expert-validated taxonomy and a 6,984-poem collection. The evaluation framework combines automatic metrics (BLEU, chrF(++), BERTScore), textual entailment, LLM-as-Judge assessments, and human qualitative analysis, revealing gaps in current models, especially for classical forms. By releasing the benchmark as open-source, the work aims to drive culturally grounded Arabic NLP research and more nuanced language understanding.

Abstract

Arabic poetry is one of the richest and most culturally rooted forms of expression in the Arabic language, known for its layered meanings, stylistic diversity, and deep historical continuity. Although large language models (LLMs) have demonstrated strong performance across languages and tasks, their ability to understand Arabic poetry remains largely unexplored. In this work, we introduce \emph{Fann or Flop}, the first benchmark designed to assess the comprehension of Arabic poetry by LLMs in 12 historical eras, covering 14 core poetic genres and a variety of metrical forms, from classical structures to contemporary free verse. The benchmark comprises a curated corpus of poems with explanations that assess semantic understanding, metaphor interpretation, prosodic awareness, and cultural context. We argue that poetic comprehension offers a strong indicator for testing how good the LLM understands classical Arabic through Arabic poetry. Unlike surface-level tasks, this domain demands deeper interpretive reasoning and cultural sensitivity. Our evaluation of state-of-the-art LLMs shows that most models struggle with poetic understanding despite strong results on standard Arabic benchmarks. We release "Fann or Flop" along with the evaluation suite as an open-source resource to enable rigorous evaluation and advancement for Arabic language models. Code is available at: https://github.com/mbzuai-oryx/FannOrFlop.

Figures (10)

• Figure 1: Chronological Wheel of Arabic Poetic Eras. This circular taxonomy visualizes the evolution of Arabic poetry across 12 major historical eras, from the Pre-Islamic and Transitional periods through the Abbasid, Andalusian, and Mamluk dynasties, up to the Modern era. The layout reflects both temporal flow and the rich cultural shifts that shaped poetic expression. Detailed taxonomy by genre, meter, and notable poets presented in Table \ref{['tab:taxonomy']}.
• Figure 2: Representative Poetic Samples Across Arabic Literary Eras. This figure presents curated excerpts from Arabic poems spanning key historical eras, illustrating the evolution of language, themes, and stylistic expression. The Pre-Islamic sample reflects tribal valor and rhetorical precision; the Umayyad excerpt captures satire and social commentary; the Abbasid example highlights philosophical reflection and refined metaphorical use; the Transitional era showcases a poetic voice confronting injustice and advocating moral clarity; and the Andalusian selection reveals emotional openness and psychological depth through lyrical expression. Together, these samples provide insight into how Arabic poetry has adapted to diverse historical, cultural, and ideological contexts. Refer to Appendix \ref{['app_en_trans']}, Figure \ref{['fig:era_sample_en']} for the GPT-4o-generated English translations of the Arabic poetic samples.
• Figure 3: Fann or Flop Pipeline. Fann or Flop is built out of the multi-stage pipeline. It begins with scraping Arabic poems from a trusted online archive using a custom web scraper. Extracted poems are matched to an initial expert-verified taxonomy and filtered to remove duplicates, ambiguous metadata, and invalid entries. The filtered texts then undergo normalization (e.g., unifying diacritics, punctuation, and letter forms) and Arabic-specific tokenization, with non-poetic or irrelevant content excluded. Manual corrections are applied to fix OCR and encoding errors. In the final stage, linguistic experts verify each sample to ensure proper alignment with genre and era labels.
• Figure 4: Qualitative Comparison of Model-Generated Explanations for a Single Arabic Poem. This figure presents a representative Arabic poem alongside its original human-written explanation and corresponding verse-by-verse explanations generated by four different language models. The comparison highlights how each model interprets the poem’s rhetorical devices, imagery, and thematic depth relative to the gold explanation. This qualitative analysis illustrates variations in faithfulness, fluency, and literary sensitivity, offering insight into each model’s ability to handle nuanced Arabic poetic language and convey its intended meaning.
• Figure 5: Era and Genre Statistics. Subfigure (a) displays the distribution of poems across historical eras, while subfigure (b) shows the overall genre distribution across the dataset.