EVALOOOP: A Self-Consistency-Centered Framework for Assessing Large Language Model Robustness in Programming

TL;DR

This work addresses the problem that traditional adversarial robustness tests fail to capture the intrinsic stability needed for autonomous coding agents. It introduces EvaLooop, a self-consistency framework that iterates between code generation and code summarization to stress-test LLMs endogenously, quantified by the Average Sustainable Loops (ASL) metric. Through large-scale evaluation of 96 LLMs on MBPP Plus, EvaLooop reveals that robustness rankings can diverge significantly from initial pass@1 performance and that adversarial perturbations provide biased or incomplete assessments. The framework is extensible to other dual tasks, demonstrated via a code translation extension, and is accompanied by an open leaderboard to guide model selection for agent-centric software engineering. These findings highlight the importance of self-consistency as a key dimension of LLM robustness with direct practical implications for reliable, autonomous software development workflows.

Abstract

Evaluating the programming robustness of large language models (LLMs) is paramount for ensuring their reliability in AI-based software development. However, adversarial attacks exhibit fundamental limitations that compromise fair robustness assessment: they demonstrate contradictory evaluation outcomes where different attack strategies tend to favor different models, and more critically, they operate solely through external perturbations, failing to capture the intrinsic stability essential for autonomous coding agents where subsequent inputs are endogenously generated by the model itself. We introduce EVALOOOP, a novel assessment framework that evaluates robustness from a self-consistency perspective, leveraging the natural duality inherent in software engineering tasks (e.g., code generation and code summarization). EVALOOOP establishes a self-contained feedback loop where an LLM iteratively transforms between code and natural language until functional failure occurs, with robustness quantified by a novel Average Sustainable Loops (ASL) metric-the mean number of iterations maintaining functional correctness across benchmark tasks. This cyclical strategy intrinsically evaluates robustness without relying on external attack configurations, providing a unified metric that reveals how effectively LLMs preserve semantic integrity through sustained self-referential transformations. We evaluate 96 popular LLMs, ranging from 0.5B to 685B parameters, on EVALOOOP equipped with the MBPP Plus benchmark, and found that EVALOOOP typically induces a 2.65%-47.62% absolute drop in pass@1 accuracy within ten loops. Intriguingly, robustness does not always align with initial performance (i.e., one-time query); for instance, Qwen3-235B-A22B-Instruct-2507, despite inferior initial code generation compared to OpenAI's o-series models and DeepSeek-V3, demonstrated the superior robustness (ASL score).

Figures (4)

• Figure 1: The generated solutions for MBPP$^+$ No. 472 task by o3-mini.
• Figure 2: An overview of EvaLooop implemented by integrating code generation and summarization loop.
• Figure 3: Distribution of average sustainable loops per task in MBPP Plus.
• Figure 4: Robustness assessment of LLMs on Framework 2.