MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark

TL;DR

MHRC-Bench provides the first multilingual, repository-level benchmark for hardware code completion across RTL, HLS, and generator-based languages, with fine-grained CST- and hardware-semantic annotations. By introducing MHRC-Bench-Train for targeted fine-tuning and MHRC-Bench-Eval for rigorous evaluation, the work demonstrates that post-training substantially boosts hardware-code completion for smaller open-weight models, and retrieval can help when properly aligned. The findings reveal distinct differences between hardware and software code completion, particularly in how structural depth and semantic categories affect performance, and highlight the importance of cross-file context and semantics in hardware design automation. Overall, MHRC-Bench is a valuable resource for advancing LLMs in hardware design, enabling finer-grained analysis and targeted improvements for EDA-era code generation.

Abstract

Large language models (LLMs) have achieved strong performance on code completion tasks in general-purpose programming languages. However, existing repository-level code completion benchmarks focus almost exclusively on software code and largely overlook hardware description languages. In this work, we present \textbf{MHRC-Bench}, consisting of \textbf{MHRC-Bench-Train} and \textbf{MHRC-Bench-Eval}, the first benchmark designed for multilingual hardware code completion at the repository level. Our benchmark targets completion tasks and covers three major hardware design coding styles. Each completion target is annotated with code-structure-level and hardware-oriented semantic labels derived from concrete syntax tree analysis. We conduct a comprehensive evaluation of models on MHRC-Bench-Eval. Comprehensive evaluation results and analysis demonstrate the effectiveness of MHRC-Bench.

Figures (23)

• Figure 1: Average EM and ES comparison of different LLMs on the MHRC-Bench-Eval benchmark.
• Figure 2: Overview of our hardware code completion benchmark across multiple hardware description paradigms. We present three representative examples from generator-based hardware construction languages (e.g., Chisel), high-level synthesis (HLS) languages (e.g., Xilinx HLS), and Register-Transfer Level (RTL) design languages (SystemVerilog). For each example, both in-file and cross-file contexts are provided, with the completion target explicitly marked. Models are required to predict the correct completion given the provided contexts, where “<TARGET>” indicates the completion position. More examples are introduced in Appendix \ref{['app:example']}
• Figure 3: Illustration of fine-grained completion target annotation in our benchmark. Completion targets are identified by aligning source code positions with Tree-sitter CST nodes and labeled with structure depth and hardware-oriented semantic categories.
• Figure 4: Impact of training data scale and completion length on hardware code completion.
• Figure 5: Effects of context length and code structure depth on hardware code completion.