Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is ChatGPT 3 safe for students?

Julia Kotovich, Manuel Oriol

TL;DR

This paper investigates whether using ChatGPT3 to generate code for student programming assignments is safe from a plagiarism-detection perspective. It employs an empirical protocol: generate Python implementations for a suite of standard data-structure and sorting algorithms, then test the produced code with Codequiry and via Google search to assess similarity to existing sources. The key finding is that, while ChatGPT3 can provide correct solutions (100% for basic prompts), a substantial portion of outputs resemble existing sources (38% detected by Codequiry, 96% on Google first-page results), leading to a strong conclusion that its use is not safe in most cases. The work highlights significant implications for educators and the limitations of current plagiarism-detection tools, and suggests expanding the evaluation to other AI tools and more complex tasks.

Abstract

ChatGPT3 is a chat engine that fulfils the promises of an AI-based chat engine: users can ask a question (prompt) and it answers in a reasonable manner. The coding-related skills of ChatGPT are especially impressive: informal testing shows that it is difficult to find simple questions that ChatGPT3 does not know how to answer properly. Some students are certainly already using it to answer programming assignments. This article studies whether it is safe for students to use ChatGPT3 to answer coding assignments (safe means that they will not be caught for plagiarism if they use it). The main result is that it is generally not safe for students to use ChatGPT3. We evaluated the safety of code generated with ChatGPT3, by performing a search with a Codequiry, a plagiarism detection tool, and searching plagiarized code in Google (only considering the first page of results). In 38% of the cases, Codequiry finds a piece of code that is partially copied by the answer of ChatGPT3. In 96% of the cases, the Google search finds a piece of code very similar to the generated code. Overall, it is not safe for students to use ChatGPT3 in 96% of the cases.

Is ChatGPT 3 safe for students?

TL;DR

This paper investigates whether using ChatGPT3 to generate code for student programming assignments is safe from a plagiarism-detection perspective. It employs an empirical protocol: generate Python implementations for a suite of standard data-structure and sorting algorithms, then test the produced code with Codequiry and via Google search to assess similarity to existing sources. The key finding is that, while ChatGPT3 can provide correct solutions (100% for basic prompts), a substantial portion of outputs resemble existing sources (38% detected by Codequiry, 96% on Google first-page results), leading to a strong conclusion that its use is not safe in most cases. The work highlights significant implications for educators and the limitations of current plagiarism-detection tools, and suggests expanding the evaluation to other AI tools and more complex tasks.

Abstract

ChatGPT3 is a chat engine that fulfils the promises of an AI-based chat engine: users can ask a question (prompt) and it answers in a reasonable manner. The coding-related skills of ChatGPT are especially impressive: informal testing shows that it is difficult to find simple questions that ChatGPT3 does not know how to answer properly. Some students are certainly already using it to answer programming assignments. This article studies whether it is safe for students to use ChatGPT3 to answer coding assignments (safe means that they will not be caught for plagiarism if they use it). The main result is that it is generally not safe for students to use ChatGPT3. We evaluated the safety of code generated with ChatGPT3, by performing a search with a Codequiry, a plagiarism detection tool, and searching plagiarized code in Google (only considering the first page of results). In 38% of the cases, Codequiry finds a piece of code that is partially copied by the answer of ChatGPT3. In 96% of the cases, the Google search finds a piece of code very similar to the generated code. Overall, it is not safe for students to use ChatGPT3 in 96% of the cases.

Paper Structure

This paper contains 10 sections, 2 tables.

Table of Contents

  1. Introduction
  2. Experiment
  3. Results
  4. Limitations and Threats to Validity
  5. Threat 1: Algorithms are not representative.
  6. Threat 2: Generated code is always the same.
  7. Threat 3: Actions are wrongly chosen to evaluate the safety of the approach.
  8. Threat 4: The study only applies to ChatGPT3.
  9. Related Work
  10. Conclusions and Future Work