Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks

Linyuan Gong, Sida Wang, Mostafa Elhoushi, Alvin Cheung

TL;DR

SAFIM presents the first large-scale, multilingual syntax-aware Fill-in-the-Middle benchmark for code. It combines three AST-based completion tasks with execution-based evaluation (and syntax matching for API calls) and a diverse set of prompts, enabling fair cross-model comparisons. The study shows that FIM pretraining improves both FIM and Left-to-Right inference and that data quality and pretraining strategies can outweigh sheer model size, highlighting the importance of pretraining design. The SAFIM benchmark and toolkit provide a foundation for future research into code-pretraining paradigms and fair evaluation methodologies.

Abstract

We introduce Syntax-Aware Fill-In-the-Middle (SAFIM), a new benchmark for evaluating Large Language Models (LLMs) on the code Fill-in-the-Middle (FIM) task. This benchmark focuses on syntax-aware completions of program structures such as code blocks and conditional expressions, and includes 17,720 examples from multiple programming languages, sourced from recent code submissions after April 2022 to minimize data contamination. SAFIM provides a robust framework with various prompt designs and novel syntax-aware post-processing techniques, facilitating accurate and fair comparisons across LLMs. Our comprehensive evaluation of 15 LLMs shows that FIM pretraining not only enhances FIM proficiency but also improves Left-to-Right (L2R) inference using LLMs. Our findings challenge conventional beliefs and suggest that pretraining methods and data quality have more impact than model size. SAFIM thus serves as a foundational platform for future research in effective pretraining strategies for code LLMs. The evaluation toolkit and dataset are available at https://github.com/gonglinyuan/safim, and the leaderboard is available at https://safimbenchmark.com.

Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks

TL;DR

SAFIM presents the first large-scale, multilingual syntax-aware Fill-in-the-Middle benchmark for code. It combines three AST-based completion tasks with execution-based evaluation (and syntax matching for API calls) and a diverse set of prompts, enabling fair cross-model comparisons. The study shows that FIM pretraining improves both FIM and Left-to-Right inference and that data quality and pretraining strategies can outweigh sheer model size, highlighting the importance of pretraining design. The SAFIM benchmark and toolkit provide a foundation for future research into code-pretraining paradigms and fair evaluation methodologies.

Abstract

We introduce Syntax-Aware Fill-In-the-Middle (SAFIM), a new benchmark for evaluating Large Language Models (LLMs) on the code Fill-in-the-Middle (FIM) task. This benchmark focuses on syntax-aware completions of program structures such as code blocks and conditional expressions, and includes 17,720 examples from multiple programming languages, sourced from recent code submissions after April 2022 to minimize data contamination. SAFIM provides a robust framework with various prompt designs and novel syntax-aware post-processing techniques, facilitating accurate and fair comparisons across LLMs. Our comprehensive evaluation of 15 LLMs shows that FIM pretraining not only enhances FIM proficiency but also improves Left-to-Right (L2R) inference using LLMs. Our findings challenge conventional beliefs and suggest that pretraining methods and data quality have more impact than model size. SAFIM thus serves as a foundational platform for future research in effective pretraining strategies for code LLMs. The evaluation toolkit and dataset are available at https://github.com/gonglinyuan/safim, and the leaderboard is available at https://safimbenchmark.com.
Paper Structure (47 sections, 5 figures, 17 tables)

This paper contains 47 sections, 5 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models for Code.
  4. Benchmarking Generative Code LLMs.
  5. Fill-in-the-Middle in Training and Evaluating Code LLMs.
  6. Benchmark Construction
  7. Corpora Collection
  8. Generating and Filtering Completion Tasks
  9. Algorithmic Block Completion.
  10. Control-Flow Completion.
  11. API Function Call Completion.
  12. Evaluation Protocols
  13. Prompts and Post-Processing
  14. Prompts
  15. Left-to-Right (L2R).
  16. ...and 32 more sections

Figures (5)

  • Figure 1: Three splits in the SAFIM benchmark illustrated with code examples. Each example includes a problem description and a code snippet, with a contiguous code segment highlighted in yellow to indicate the part to be masked and completed by LLMs. Contexts in these examples are shortened for clarity.
  • Figure 2: The original code is shown in the top-left, with the block a, b = b, a + b to be masked. The subsequent cells illustrate five distinct prompt types. The "$\triangleleft$" symbol indicates the end of the prompt, where model generation begins. The tokens [MASK] and [END] are model-specific, e.g., <SUF> and <MID> for CodeLLaMa, and <|mask:0|> and <|mask:1|> for InCoder.
  • Figure 3: Average performance of different models relative to their sizes on the SAFIM benchmark. Each model is represented by a dot, with the x-axis showing model size (number of parameters) and the y-axis showing average performance across three task categories. Dot colors signify pretraining paradigms: red for Left-to-Right (L2R), blue for FIM, purple for a combination of L2R and FIM, and orange for proprietary models with undisclosed pretraining methods.
  • Figure 4: Histogram of the total number of characters of the natural language problem description and the code context. 424 example longer than 8,000 characters are excluded from this histogram for clarity but counted towards the displayed quantiles.
  • Figure 5: Pass@1 scores for each model on algorithmic block completion across various months in the new test dataset.