RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian

TL;DR

RoCode introduces the first Romanian-language benchmark for code intelligence by collecting 2,642 Romanian problem statements with 11k solutions in C/C++ and Python, accompanied by extensive test suites. The dataset targets evaluation and fine-tuning of Romanian- and multilingual-language models on NL-PL tasks, highlighting the scarcity of non-English code data and the challenge of code-generation from Romanian prompts. Experimental results show Romanian models perform near zero on RoCode, while English-oriented models achieve limited nonzero performance, underscoring the need for dedicated Romanian code-data pipelines and instruction-tuning. The work also discusses dataset design decisions, code-code-switching phenomena, and directions for future multilingual code intelligence research.

Abstract

Recently, large language models (LLMs) have become increasingly powerful and have become capable of solving a plethora of tasks through proper instructions in natural language. However, the vast majority of testing suites assume that the instructions are written in English, the de facto prompting language. Code intelligence and problem solving still remain a difficult task, even for the most advanced LLMs. Currently, there are no datasets to measure the generalization power for code-generation models in a language other than English. In this work, we present RoCode, a competitive programming dataset, consisting of 2,642 problems written in Romanian, 11k solutions in C, C++ and Python and comprehensive testing suites for each problem. The purpose of RoCode is to provide a benchmark for evaluating the code intelligence of language models trained on Romanian / multilingual text as well as a fine-tuning set for pretrained Romanian models. Through our results and review of related works, we argue for the need to develop code models for languages other than English.

Figures (5)

• Figure 1: Example Romanian problem statement from RoCode. Problems specify a story-like description followed by input and output data specifications and restrictions. Finally, an example of an output from a correct solution is provided. We also show the English translation for non-Romanian readers.
• Figure 2: Percentage of Romanian variable names and function names that contain explicit Romanian words in submitted human C / C++ solutions. Abbreviations are omitted, making the actual proportion larger than shown here.
• Figure 3: Distribution of problem difficulties in RoCode. Problem difficulty is estimated automatically based on the number of correct solutions, unique users, and the date of the problem.
• Figure 4: Correlation between problem statement length and solution lengths across problem difficulties. Harder problems require longer solutions, and problem statement length is not correlated with solution length.
• Figure 5: Percentage of Romanian variable and function names that contain explicit Romanian words in model-generated Python solutions. Generated solutions tend to have more explicit Romanian names.