The teaching goal of the writing task is not only to complete an essay, but more importantly, to equip students with specific knowledge and skills in the creative process. However, the widespread use of generative AI tools is a concern for teachers. Teachers are concerned that students will treat the content produced by AI tools as their own original work. In this case, not only does the student fail to achieve the real learning goals, but it may even affect academic integrity.
In order to solve this problem, AI detection tools came into being. The AI detection tool can detect the text generated by generative AI, which can be used as a reference for teachers to provide further guidance. This report tests 16 of the most popular AI detection tools on the market. The test results show that most AI detection tools are able to recognize GPT-35 generated text, but it cannot effectively recognize the text generated by GPT-4. However, Copyleaks, Turnitin, and OriginalityThese three AI tools have shown high accuracy in detecting GPT-4. Here's how the test works:
1.Select an AI detection tool
This report selects 16 of the most popular AI detection tools on the market, including:
2.Prepare the test text
A total of 126 test texts were selected and divided into three groups for testing. The first group consists of 42 articles written by first-year students during the 2014-2015 academic year, which were completed before generative AI tools became widespread, ensuring that the articles were not generated by AI. The second group is 42 articles by GPT-35 generated**. The third group consists of 42 articles generated by GPT-4. The test texts for the second and third groups were generated in the first week of April 2023 and covered a variety of fields such as social sciences, natural sciences, and humanities.
3.Take the test
The test was conducted from June 25, 2023 to July 12, 2023. All test texts have been removed from **, bullets, etc., and submitted to the 16 detectors in plain text format. The test results for each article are categorized: AI-generated, human-written, or inconclusive. Among them, AI-generated means that most of the text may be generated by AI, but it does not necessarily mean that the entire text is AI-generated.
4.Compare the results
42 test results of students**).
Student** group: Copyleaks, Turnitin, GPT Radar, and ContentDetector had the highest accuracy and the lowest false positive rates. originality.The accuracy rate of 9 tools, including AI and Scribbr, is more than 85%. seo.AI, SAPLING, and ZeroGPT have higher false positive rates.
42 GPT-35. Generate the detection results of the article).
gpt-3.Group 5: Most of the tools are able to recognize GPT-35. The content generated has an accuracy rate of more than 86%. However, the accuracy of CrossPlag, Content at Scale, and ContentDetector was low and did not reach the average of the tests in this group.
42 GPT-40 test results for production articles).
gpt-4.Group 0: Face GPT-40 generated text, only copyleaks, turnitin, and originalityThe accuracy of AI is high. The results of the rest of the testing tools are not stable and the false positive rate is also high. Arguably, this is the most important difference between these 3 and 13 other detection tools.
126 test texts).
Based on the results of 126 test texts, the 16 test tools can be divided into three levels. The first tranche is Copyleaks, Turnitin, and OriginalityAI, with an accuracy rate of more than 90%, is excellent. The second gear is other tools such as scribbr, zerogpt, grammica, etc., with an accuracy rate between 63% and 88%. The third gear is sapling and contentdector, with an accuracy rate of less than 63%.
5.Conclusion
Most tests can detect GPT-35 ** and human-written text, but cannot effectively detect GPT-4 generated content. However, Copyleaks, Turnitin, and OriginalityThe AI also showed high accuracy when it came to detecting content generated by GPT-4.
In teaching scenarios where the use of AI tools is allowed, it is necessary for teachers to confirm the general situation of students' use of AI tools to avoid students abusing AI tools. Considering the false positives in AI detection tools, the results of AI detection tools should not be used as the sole criterion for judging academic misconduct. In order to protect the interests of students, teachers need to analyze the specific situation and consider it comprehensively based on students' daily homework, school policies and other factors.
References:the effectiveness of software designed to detect ai-generated writing: a comparison of 16 ai text detectors
February** Dynamic Incentive Program