Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Code Generation of LLMs in Advanced Computer Science Problems

Emir Catir, Robin Claesson, Rodothea Myrsini Tsoupidi

TL;DR

This study evaluates four publicly accessible LLM-based code-generation tools across Java, Python, and C on 12 problems (3 introductory baselines and 9 advanced CS problems). By generating code and verifying it with a dedicated 1000-test-case suite per problem, the work shows that LLMs are highly capable on introductory tasks but struggle with advanced assignments, often producing partial solutions or heuristics rather than fully correct implementations. Copilot generally yields the strongest results among the tools, but overall accuracy on advanced problems remains limited, with notable exceptions and clear failure modes linked to algorithmic constraints and input handling. The findings offer practical guidance for CS educators on designing assignments that assess higher-order problem-solving and constraint satisfaction, as well as highlighting the need for targeted prompt engineering and robust verification in LLM-assisted programming education.

Abstract

Large Language Models (LLMs), such as GitHub Copilot and ChatGPT have become popular among programming students. Students use LLMs to assist them in programming courses, including generating source code. Previous work has evaluated the ability of LLMs in solving introductory-course programming assignments. The results have shown that LLMs are highly effective in generating code for introductory Computer Science (CS) courses. However, there is a gap in research on evaluating LLMs' ability to generate code that solves advanced programming assignments. In this work, we evaluate the ability of four LLM tools to solve programming assignments from advanced CS courses in three popular programming languages, Java, Python, and C. We manually select 12 problems, three problems from introductory courses as the baseline and nine programming assignments from second- and third-year CS courses. To evaluate the LLM-generated code, we generate a test suite of 1000 test cases per problem and analyze the program output. Our evaluation shows that although LLMs are highly effective in generating source code for introductory programming courses, solving advanced programming assignments is more challenging. Nonetheless, in many cases, LLMs identify the base problem and provide partial solutions that may be useful to CS students. Furthermore, our results may provide useful guidance for teachers of advanced programming courses on how to design programming assignments.

Evaluating Code Generation of LLMs in Advanced Computer Science Problems

TL;DR

This study evaluates four publicly accessible LLM-based code-generation tools across Java, Python, and C on 12 problems (3 introductory baselines and 9 advanced CS problems). By generating code and verifying it with a dedicated 1000-test-case suite per problem, the work shows that LLMs are highly capable on introductory tasks but struggle with advanced assignments, often producing partial solutions or heuristics rather than fully correct implementations. Copilot generally yields the strongest results among the tools, but overall accuracy on advanced problems remains limited, with notable exceptions and clear failure modes linked to algorithmic constraints and input handling. The findings offer practical guidance for CS educators on designing assignments that assess higher-order problem-solving and constraint satisfaction, as well as highlighting the need for targeted prompt engineering and robust verification in LLM-assisted programming education.

Abstract

Large Language Models (LLMs), such as GitHub Copilot and ChatGPT have become popular among programming students. Students use LLMs to assist them in programming courses, including generating source code. Previous work has evaluated the ability of LLMs in solving introductory-course programming assignments. The results have shown that LLMs are highly effective in generating code for introductory Computer Science (CS) courses. However, there is a gap in research on evaluating LLMs' ability to generate code that solves advanced programming assignments. In this work, we evaluate the ability of four LLM tools to solve programming assignments from advanced CS courses in three popular programming languages, Java, Python, and C. We manually select 12 problems, three problems from introductory courses as the baseline and nine programming assignments from second- and third-year CS courses. To evaluate the LLM-generated code, we generate a test suite of 1000 test cases per problem and analyze the program output. Our evaluation shows that although LLMs are highly effective in generating source code for introductory programming courses, solving advanced programming assignments is more challenging. Nonetheless, in many cases, LLMs identify the base problem and provide partial solutions that may be useful to CS students. Furthermore, our results may provide useful guidance for teachers of advanced programming courses on how to design programming assignments.

Paper Structure

This paper contains 20 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluating LLM in CS courses.
  4. Evaluating LLM in introductory programming assignments.
  5. Evaluating LLM in advanced programming assignments.
  6. Motivating Example
  7. Methodology
  8. LLM Selection
  9. Problem Selection
  10. Code Generation
  11. Code Verification
  12. Evaluation
  13. RQ1: How effective are LLM tools at solving advanced programming assignments correctly?
  14. Summary
  15. RQ2: What is the capability of LLM tools at identifying the problem to solve?
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Problem description prompt for the boxes problem (Table \ref{['tab:problems']}).
  • Figure 2: Code Generation and Verification
  • Figure 3: Cow Game: Generated code by Github Copilot in C and corrected code.

Theorems & Definitions (2)

  • definition 1: Accuracy
  • definition 2: Correctness