OmniEvalKit: A Modular, Lightweight Toolbox for Evaluating Large Language Model and its Omni-Extensions

TL;DR

The paper addresses the challenge of comprehensively evaluating LLMs and their omni-extensions across multilingual, multidomain, and multimodal tasks. It introduces OmniEvalKit, a modular, lightweight benchmarking toolbox with a two-layer architecture (Static Builder and Dynamic Data Flow) that decouples model construction from data flow to enable rapid integration of models and datasets. The framework supports over 100 LLMs and 50 evaluation datasets, provides components for answer extraction, flexible generation and decoding, and customizable metrics through a unified JSON data format, enabling thousands of model–dataset evaluations. This approach facilitates open evaluation, supports cross-domain and cross-modal benchmarking, and enables downstream research such as scaling laws and vertical-domain model selection, with demonstrated stability across diverse hardware and a path toward broader deployment.

Abstract

The rapid advancements in Large Language Models (LLMs) have significantly expanded their applications, ranging from multilingual support to domain-specific tasks and multimodal integration. In this paper, we present OmniEvalKit, a novel benchmarking toolbox designed to evaluate LLMs and their omni-extensions across multilingual, multidomain, and multimodal capabilities. Unlike existing benchmarks that often focus on a single aspect, OmniEvalKit provides a modular, lightweight, and automated evaluation system. It is structured with a modular architecture comprising a Static Builder and Dynamic Data Flow, promoting the seamless integration of new models and datasets. OmniEvalKit supports over 100 LLMs and 50 evaluation datasets, covering comprehensive evaluations across thousands of model-dataset combinations. OmniEvalKit is dedicated to creating an ultra-lightweight and fast-deployable evaluation framework, making downstream applications more convenient and versatile for the AI community.

Figures (1)

• Figure 1: Illustration and API Example of OmniEvalKit.