Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs Still Can't Avoid Instanceof: An Investigation Into GPT-3.5, GPT-4 and Bard's Capacity to Handle Object-Oriented Programming Assignments

Bruno Pereira Cipriano, Pedro Alves

TL;DR

This study evaluates GPT-3.5, GPT-4, and Bard on six real-world object-oriented programming assignments used in a CS education context, using the Drop Project Automatic Assessment Tool (AAT) for evaluation. It finds that while all models can produce mostly working code, they frequently violate OO best practices (e.g., relying on instanceof) and require guided prompting, with GPT-4 showing the strongest performance. The paper provides a detailed analysis of model-specific errors, prompts used, and improvements, highlighting the potential and limits of LLM-assisted learning in OO design. It advocates integrating LLMs with AATs and adopting project-based, code-quality–oriented pedagogy to safely leverage these tools in classrooms.

Abstract

Large Language Models (LLMs) have emerged as promising tools to assist students while solving programming assignments. However, object-oriented programming (OOP), with its inherent complexity involving the identification of entities, relationships, and responsibilities, is not yet mastered by these tools. Contrary to introductory programming exercises, there exists a research gap with regard to the behavior of LLMs in OOP contexts. In this study, we experimented with three prominent LLMs - GPT-3.5, GPT-4, and Bard - to solve real-world OOP exercises used in educational settings, subsequently validating their solutions using an Automatic Assessment Tool (AAT). The findings revealed that while the models frequently achieved mostly working solutions to the exercises, they often overlooked the best practices of OOP. GPT-4 stood out as the most proficient, followed by GPT-3.5, with Bard trailing last. We advocate for a renewed emphasis on code quality when employing these models and explore the potential of pairing LLMs with AATs in pedagogical settings. In conclusion, while GPT-4 showcases promise, the deployment of these models in OOP education still mandates supervision.

LLMs Still Can't Avoid Instanceof: An Investigation Into GPT-3.5, GPT-4 and Bard's Capacity to Handle Object-Oriented Programming Assignments

TL;DR

This study evaluates GPT-3.5, GPT-4, and Bard on six real-world object-oriented programming assignments used in a CS education context, using the Drop Project Automatic Assessment Tool (AAT) for evaluation. It finds that while all models can produce mostly working code, they frequently violate OO best practices (e.g., relying on instanceof) and require guided prompting, with GPT-4 showing the strongest performance. The paper provides a detailed analysis of model-specific errors, prompts used, and improvements, highlighting the potential and limits of LLM-assisted learning in OO design. It advocates integrating LLMs with AATs and adopting project-based, code-quality–oriented pedagogy to safely leverage these tools in classrooms.

Abstract

Large Language Models (LLMs) have emerged as promising tools to assist students while solving programming assignments. However, object-oriented programming (OOP), with its inherent complexity involving the identification of entities, relationships, and responsibilities, is not yet mastered by these tools. Contrary to introductory programming exercises, there exists a research gap with regard to the behavior of LLMs in OOP contexts. In this study, we experimented with three prominent LLMs - GPT-3.5, GPT-4, and Bard - to solve real-world OOP exercises used in educational settings, subsequently validating their solutions using an Automatic Assessment Tool (AAT). The findings revealed that while the models frequently achieved mostly working solutions to the exercises, they often overlooked the best practices of OOP. GPT-4 stood out as the most proficient, followed by GPT-3.5, with Bard trailing last. We advocate for a renewed emphasis on code quality when employing these models and explore the potential of pairing LLMs with AATs in pedagogical settings. In conclusion, while GPT-4 showcases promise, the deployment of these models in OOP education still mandates supervision.
Paper Structure (17 sections, 1 figure, 1 table)

This paper contains 17 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. The assignments
  4. Methodology
  5. It works but it's not OO - how LLM(s) tried to solve the IT Company assignment
  6. Main analysis
  7. Attempts to improve the LLM(s)' solutions
  8. Overall Findings
  9. Problem categorization
  10. Recommendations for CS Educators
  11. Give more weight to code quality evaluations
  12. Use LLMs in your classes
  13. Adopt project-based learning
  14. On the importance of citing your sources
  15. Limitations
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Simplified UML class diagram for the "IT Company" assignment, excluding non-essential attributes, methods, and constructors. Students only receive textual instructions, not this diagram.