MERA: A Comprehensive LLM Evaluation in Russian

TL;DR

MERA presents an open, Russian-language evaluation benchmark for large language and foundation models, featuring 21 instruction-form datasets across 10 skills to assess natural language understanding, reasoning, coding, and ethics. The framework emphasizes a fixed, leakage-resistant evaluation with zero- and few-shot settings, supported by an open-codebase and leaderboard, and includes both open-model baselines and human baselines. Results indicate current open LLMs lag behind human capabilities on many tasks, particularly in ethical diagnostics, while some models excel on math, logic, and world-knowledge questions. The work plans to extend MERA to multimodal modalities (images, audio) and invites community contributions to broaden task coverage and robustness of Russian-language evaluation.

Abstract

Over the past few years, one of the most notable advancements in AI research has been in foundation models (FMs), headlined by the rise of language models (LMs). As the models' size increases, LMs demonstrate enhancements in measurable aspects and the development of new qualitative features. However, despite researchers' attention and the rapid growth in LM application, the capabilities, limitations, and associated risks still need to be better understood. To address these issues, we introduce an open Multimodal Evaluation of Russian-language Architectures (MERA), a new instruction benchmark for evaluating foundation models oriented towards the Russian language. The benchmark encompasses 21 evaluation tasks for generative models in 11 skill domains and is designed as a black-box test to ensure the exclusion of data leakage. The paper introduces a methodology to evaluate FMs and LMs in zero- and few-shot fixed instruction settings that can be extended to other modalities. We propose an evaluation methodology, an open-source code base for the MERA assessment, and a leaderboard with a submission system. We evaluate open LMs as baselines and find that they are still far behind the human level. We publicly release MERA to guide forthcoming research, anticipate groundbreaking model features, standardize the evaluation procedure, and address potential societal drawbacks.

Figures (1)

• Figure 1: The MERA benchmark illustration. The benchmark incorporates 21 tasks covering 10 skills within an assessment platform with a fixed experimental pipeline for LLM evaluation for the Russian language.