CLARC: C/C++ Benchmark for Robust Code Search

TL;DR

An automated pipeline for code search datasets and CLARC, a C/C++ benchmark built from real-world GitHub repositories, are introduced and evaluation of six state-of-the-art models reveals sharp drops in retrieval effectiveness.

Abstract

Efficient code retrieval is critical for developer productivity, yet existing benchmarks largely focus on Python and rarely stress-test robustness beyond superficial lexical cues. To address the gap, we introduce an automated pipeline for code search datasets and present CLARC, a C/C++ benchmark built from real-world GitHub repositories. CLARC contains 1,245 query-code pairs for evaluation and 5,472 pairs for training. The benchmark incorporates LLM-generated natural language queries validated through rigorous human scoring and hypothesis testing. To analyze contextual requirements effectively, our pipeline starts by ensuring code compilability. It then categorizes code snippets by dependency complexity, distinguishing whether the code relies on custom-defined types or helper functions. The pipeline also enables CLARC to stress-test retrieval robustness by introducing challenging settings, including identifier anonymization and compilation to low-level languages like Assembly and WebAssembly. Under these conditions, our evaluation of six state-of-the-art models reveals sharp drops in retrieval effectiveness. The experimental results highlight the models' persistent reliance on lexical features rather than code semantic understanding. Our dataset is publicly available at https://huggingface.co/datasets/ClarcTeam/CLARC.

Figures (5)

• Figure 1: Example Functions of CLARC. The full examples with their corresponding queries can be found in Appendix \ref{['appendix: examples']}.
• Figure 2: Prompt for Group 1
• Figure 3: Prompt for Group 2
• Figure 4: Prompt for Group 3
• Figure 5: Prompt for Style Alignment