BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism

TL;DR

BenCzechMark addresses the lack of a fair, Czech-centric benchmark for evaluating few-shot LLMs by introducing a multitask, multimetric benchmark with a statistically grounded duel scoring mechanism. The framework combines native Czech data with selective translations, multiple task formats, and threshold-free metrics (AUROC) to mitigate calibration biases, while using significance testing to guard against chance improvements. The authors also create a large Czech corpus (BUT-LCC) for contamination analysis and continuous pretraining of Czech-centric models, and provide a public leaderboard to enable ongoing, robust comparisons. Overall, BCM advances equitable Czech NLP benchmarking and highlights the challenges of monolingual Czech model development compared to multilingual baselines, offering a foundation for future multilingual adaptation and data augmentation.

Abstract

We present BenCzechMark (BCM), the first comprehensive Czech language benchmark designed for large language models, offering diverse tasks, multiple task formats, and multiple evaluation metrics. Its duel scoring system is grounded in statistical significance theory and uses aggregation across tasks inspired by social preference theory. Our benchmark encompasses 50 challenging tasks, with corresponding test datasets, primarily in native Czech, with 14 newly collected ones. These tasks span 8 categories and cover diverse domains, including historical Czech news, essays from pupils or language learners, and spoken word. Furthermore, we collect and clean BUT-Large Czech Collection, the largest publicly available clean Czech language corpus, and use it for (i) contamination analysis and (ii) continuous pretraining of the first Czech-centric 7B language model with Czech-specific tokenization. We use our model as a baseline for comparison with publicly available multilingual models. Lastly, we release and maintain a leaderboard with existing 50 model submissions, where new model submissions can be made at https://huggingface.co/spaces/CZLC/BenCzechMark.

Figures (9)

• Figure 2: Model size (in B of parameters) vs Average Win score for models smaller (a) and larger (b) than 16B parameters. Marker $\bullet$ denotes the instructed model, whereas $\blacktriangle$ denotes the base model trained for the next token prediction. The marker size is scaled proportionally to the model's max-centered variance. Here, model family refers to a pair of a base model and its corresponding instruction-tuned variant.
• Figure 3: Average Max-centered Variance.
• Figure 4: Category Win Score of models under (a) or over (b) 16B parameters. Categories are: Czech Math Reasoning (CMR), Language Modeling (LM), Named Entity Recognition (NER), Reading Comprehension (RC) Factual Knowledge (FK), Czech Language Understanding (CLU), Sentiment Analysis (S), and Natural Language Inference (NLI).
• Figure 7: Generated response by Llama 3.2 Instruct 3B for a short newspaper article. The system here mismatched velvet revolution (sametová revoluce) for dissolution of Austo-Hungary, and invented English-like non-Czech words including their declension forms (red, underlined).
• Figure 8: Duel Win Scores for every task and every model under 16B (a) and over 16B (b) of parameters.