Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs' Instruction Following Capability

TL;DR

This work introduces Meeseeks, a feedback-driven, iterative benchmark for evaluating LLMs' instruction-following under multi-constraint prompts. It combines 32 capability tags across three cognitive dimensions with a code-guided, rule-augmented evaluation pipeline to automatically identify constraint violations and guide self-correction over 20 turns. Across 17 models, even after extensive feedback, most fail to exceed 90% accuracy, revealing substantial ceilings and highlighting the divergence between single-turn performance and upper-bound instruction-following capability. The results demonstrate significant gains in evaluation accuracy and token efficiency through code-guided evaluation, and offer crucial insights into model limitations and future directions for robust instruction-following systems.

Abstract

The capability to precisely adhere to instructions is a cornerstone for Large Language Models (LLMs) to function as dependable agents in real-world scenarios. However, confronted with complex prompts, LLMs frequently encounter difficulties in fulfilling all specified requirements within a single response. Drawing inspiration from recent advancements in Chain-of-Thought (CoT) prompting and self-correction methodologies, we introduce Meeseeks (The name is inspired by Mr. Meeseeks from "Rick and Morty," a character renowned for efficiently accomplishing assigned tasks. See: https://en.wikipedia.org/wiki/Mr._Meeseeks), a fully automated iterative instruction-following benchmark equipped with an integrated feedback mechanism. Meeseeks identifies erroneous components in model responses and provides corresponding feedback accurately, thereby iteratively guiding the model toward self-correction. The dataset contains over 700 curated instances annotated by 32 distinct capability tags in Chinese and English. Extensive experimental results reveal that different state-of-the-art commercial and open-source LLMs exhibit vastly disparate performance, and even after 20 turns of iterative feedback-driven self-correction, nearly all models demonstrate suboptimal performance. We conducted comprehensive analysis from both macro and instance levels, uncovering numerous common issues prevalent in current state-of-the-art models, as well as several counterintuitive phenomena. We've open-sourced our work on https://github.com/ADoublLEN/Meeseeks.

Figures (7)

• Figure 1: One complete Meeseeks iteration includes response collection, evaluation, and feedback prompting.
• Figure 2: Utility rate over 20 turns (Complete result in Appendix \ref{['sec:complete_experiments_results']})
• Figure 3: Our code-guided formatting demonstrates remarkably consistent performance in both token utilization and accuracy metrics.
• Figure 4: As the turns iterate, the standard deviation of utility rates gradually increases; The correlation between subsequent round rankings and turn 1 rankings shows a declining trend.
• Figure 5: Performance trends varied significantly across models and capability tags.