An Evaluation of Current Trends in AI-Generated Text in Otolaryngology Publications.
Access to this document is restricted. Some items have been embargoed at the request of the author, but will be made publicly available after the "No Access Until" date.
During the embargo period, you may request access to the item by clicking the link to the restricted file(s) and completing the request form. If we have contact information for a Cornell author, we will contact the author and request permission to provide access. If we do not have contact information for a Cornell author, or the author denies or does not respond to our inquiry, we will not be able to provide access. For more information, review our policies for restricted content.
OBJECTIVES: Since the release of ChatGPT-4 in March 2023, large language models (LLMs) application in biomedical manuscript production has been widespread. GPT-modified text detectors, such as GPTzero, lack sensitivity and reliability and do not quantify the amount of AI-generated text. However, recent work has identified certain adjectives more frequently used by LLMs that can help identify and quantify LLM-modified text. The aim of this study is to utilize these adjectives to identify LLM-generated text in otolaryngology publications. STUDY DESIGN: Meta-research. METHODS: Twenty-five otolaryngology journals were studied between November 2022 and July 2024, encompassing 8751 published works. Articles from countries where ChatGPT-4 is not available were removed, yielding 7702 articles for study inclusion. These publications were analyzed using a Python script to determine the frequency of the top 100 adjectives disproportionately generated by ChatGPT-4. RESULTS: A significant increase in the frequency of adjectives associated with GPT use was observed from November 2023 to July 2024 across all journals (p < 0.001), with a significant difference before and after the release of ChatGPT in March 2023. Journals with higher impact factors had significantly lower usage of GPT-associated adjectives than those with lower impact factors (p < 0.001). There was no significant difference in GPT-associated adjective use by first authors with a doctoral degree versus those without. Publications by authors from English-speaking countries demonstrated a significantly more frequent use of LLM-associated adjectives (p < 0.001). CONCLUSIONS: This study suggests that ChatGPT use in otolaryngology manuscript production has significantly increased since the release of ChatGPT-4. Future research should be aimed at further characterizing the landscape of AI-generated text in otolaryngology and developing tools that encourage authors' transparency regarding the use of LLMs. LEVEL OF EVIDENCE: NA.