Is ChatGPT equal to humans in writing the introduction sections to scientific articles? That's the question posed by this week's article in the Medcopywriter Journal Club.
Citation
Sikander B, et al. ChatGPT-4 and Human Researchers Are Equal in Writing Scientific Introduction Sections: A Blinded, Randomized, Non-inferiority Controlled
Study. Cureus. 2023;15(11):e49019.
Overview
Generative AI is rapidly advancing. ChatGPT has been around for more than 12 months now, but how good is it getting at medical writing? In a recent paper published in the journal Cureus, Sikander and colleagues from Odense University Hospital carried out a randomised non-inferiority study to find out.
ChatGPT-4 was asked to write 18 introduction sections based on the aim of previously published studies, and these sections were compared to the human-written introductions already published in a medical journal. The outputs were then compared by eight blinded assessors (with PhDs).
Summary
There was no significant difference between ChatGPT and human introductions regarding publishability, readability and content quality
Most of the assessors (59%) preferred ChatGPT-written introductions, whereas 33% preferred human-written content
Less than half of the assessors half could say which were written by GPT-4 or humans
ChatGPT sentences were found to be longer and had longer words
Key quote
"GPT-4 can be a valuable tool in writing introduction sections, but the use would require research-based background knowledge to ensure the validity of the output."
Opinion
I think this is a really interesting study - ChatGPT was found not just to be equally as good as humans at writing introductions for scientific articles, but the content was frequently preferred over the human-written introductions
Writing the introductory section to a paper can be a laborious and time-consuming task, so having help from AI could be a massive timesaver
It's important, of course, that any content created by AI is checked for accuracy, relevance and plagiarism, but I still think it can be used to speed up the writing process
It should be noted that although ChatGPT had a significantly better readability score, this small difference was not considered relevant
A key point missing is referencing. Asking ChatGPT to write an introduction is one thing, but getting it to provide accurate sources is currently beyond its capabilities. Indeed, the authors state, "We removed references from the original articles as GPT-4's output does not automatically include references, and also since this was beyond the scope of this study."
Interestingly, in the appendix, you can read the prompts used to create the scientific introductions, as well as the answers