Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving

Md Sifat Hossain, Anika Tabassum, Md. Fahim Arefin, Tarannum Shaila Zaman

TL;DR

ICPC-style problems provide a rigorous benchmark for evaluating LLMs in algorithmic reasoning and code generation. The paper introduces LLM-ProS, a four-stage evaluation framework that uses 166 World Finals problems (2011–2024) to compare GPT-4o, Mistral Large, Llama-3.1-405B, o1-mini, and o1-preview on reasoning, accuracy, and efficiency, while examining dataset contamination and chain-of-thought prompting. Key findings show that o1 models significantly outperform others on unseen problems and do so more efficiently, underscoring the value of CoT training and contamination-free benchmarks for reliable technical problem solving. The work informs LLM design and evaluation strategies for competitive programming and related algorithmic tasks, highlighting practical implications for robust, resource-conscious AI systems.

Abstract

The rapid advancement of large language models has opened new avenues for automating complex problem-solving tasks such as algorithmic coding and competitive programming. This paper introduces a novel evaluation technique, LLM-ProS, to assess the performance of state-of-the-art LLMs on International Collegiate Programming Contest (ICPC) problems. Using a curated dataset of 166 World Finals problems from 2011 to 2024, we benchmark the models' reasoning, accuracy, and efficiency. We evaluate the five models-GPT-4o, Mistral Large, Llama-3.1-405B, and the o1 family, consisting of o1-mini and o1-preview, across critical metrics like correctness, resource utilization, and response calibration. Our results reveal significant differences in the models' abilities to generalize, adapt, and solve novel problems. We also investigated the impact of training methodologies, dataset contamination, and chain-of-thought reasoning on model performance. The findings provide new insights into optimizing LLMs for algorithmic tasks, highlighting both strengths and limitations of current models.

LLM-ProS: Analyzing Large Language Models' Performance in Competitive Problem Solving

TL;DR

ICPC-style problems provide a rigorous benchmark for evaluating LLMs in algorithmic reasoning and code generation. The paper introduces LLM-ProS, a four-stage evaluation framework that uses 166 World Finals problems (2011–2024) to compare GPT-4o, Mistral Large, Llama-3.1-405B, o1-mini, and o1-preview on reasoning, accuracy, and efficiency, while examining dataset contamination and chain-of-thought prompting. Key findings show that o1 models significantly outperform others on unseen problems and do so more efficiently, underscoring the value of CoT training and contamination-free benchmarks for reliable technical problem solving. The work informs LLM design and evaluation strategies for competitive programming and related algorithmic tasks, highlighting practical implications for robust, resource-conscious AI systems.

Abstract

The rapid advancement of large language models has opened new avenues for automating complex problem-solving tasks such as algorithmic coding and competitive programming. This paper introduces a novel evaluation technique, LLM-ProS, to assess the performance of state-of-the-art LLMs on International Collegiate Programming Contest (ICPC) problems. Using a curated dataset of 166 World Finals problems from 2011 to 2024, we benchmark the models' reasoning, accuracy, and efficiency. We evaluate the five models-GPT-4o, Mistral Large, Llama-3.1-405B, and the o1 family, consisting of o1-mini and o1-preview, across critical metrics like correctness, resource utilization, and response calibration. Our results reveal significant differences in the models' abilities to generalize, adapt, and solve novel problems. We also investigated the impact of training methodologies, dataset contamination, and chain-of-thought reasoning on model performance. The findings provide new insights into optimizing LLMs for algorithmic tasks, highlighting both strengths and limitations of current models.

Paper Structure

This paper contains 33 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Motivation
  4. Background
  5. GPT-4o
  6. Mistral Large
  7. Llama-3.1-405B
  8. OpenAI o1 Family
  9. Methodology
  10. Data Collection
  11. Data Preprocessing
  12. Model Testing
  13. Solution Generation and Submission
  14. Experimental Evaluation
  15. Evaluation Metrics
  16. ...and 18 more sections

Figures (4)

  • Figure 1: The Overview of LLM-ProS
  • Figure 2: Accuracy Trends Across Years (Heatmap)
  • Figure 3: Number of solved problems across categories.
  • Figure 4: Verdict Distribution Across Models