NotSoTiny: A Large, Living Benchmark for RTL Code Generation

TL;DR

NotSoTiny tackles the challenge of evaluating LLMs on realistic RTL code generation by building a large, contamination-resilient, context-rich benchmark from hundreds of Tiny Tapeout designs. The authors implement a fully automated pipeline to produce 1,114 validated contextual-module completion tasks (from an initial 3,062 unique candidates) and assess outputs with scalable formal equivalence checking, moving beyond traditional syntax and testbench tests. Their results show that state-of-the-art LLMs achieve only about $20\%$ functional correctness under equivalence checking, despite high syntax validity, underscoring the gap between surface-level correctness and full behavioral equivalence. The work also demonstrates contamination-control methods and introduces a living benchmark paradigm, enabling periodic updates to stay ahead of model pretraining data and guiding future improvements in RTL-code generation.

Abstract

LLMs have shown early promise in generating RTL code, yet evaluating their capabilities in realistic setups remains a challenge. So far, RTL benchmarks have been limited in scale, skewed toward trivial designs, offering minimal verification rigor, and remaining vulnerable to data contamination. To overcome these limitations and to push the field forward, this paper introduces NotSoTiny, a benchmark that assesses LLM on the generation of structurally rich and context-aware RTL. Built from hundreds of actual hardware designs produced by the Tiny Tapeout community, our automated pipeline removes duplicates, verifies correctness and periodically incorporates new designs to mitigate contamination, matching Tiny Tapeout release schedule. Evaluation results show that NotSoTiny tasks are more challenging than prior benchmarks, emphasizing its effectiveness in overcoming current limitations of LLMs applied to hardware design, and in guiding the improvement of such promising technology.

Figures (5)

• Figure 1: Overview of the NotSoTiny benchmark construction pipeline. Designs from multiple Tiny Tapeout shuttles are filtered, merged, converted into contextual-module completion tasks, and deduplicated to produce a clean, diverse, and contamination-resilient benchmark.
• Figure 2: Average Min-K computed for different values of K (i.e., the K% of tokens with lowest probability) around the value $K=20$ recommended by the authors of Min-K. Results are aggregated across all samples and models under study. Exponential is taken for easier interpretability with the Area Under the Curve (AUC), and thanks to its monotonicity, it doesn't affect the comparison. Higher values indicate higher chances of contamination.
• Figure 3: Syntax correctness (STX) and testbench correctness (FNC-Test) for three representative LLMs across four RTL benchmarks. Results show large drops from STX to FNC-Test on VerilogEval, RTLLM, and CVDP. Formal equivalence correctness (FNC-EQV) is also included for the NotSoTiny benchmark
• Figure 4: Coverage and assertion analysis of Tiny Tapeout testbenches. Each point represents a project, showing code coverage versus number of assertions.
• Figure 5: TuRTLe Leaderboard showing model performance on the NotSoTiny-25-12 benchmark for Module Completion. Available at: https://huggingface.co/spaces/HPAI-BSC/TuRTLe-Leaderboard.