Study estimates 13% of biomedical abstracts published in 2024 involved use of AI – OXBIG NEWS NETWORK-OxBig News Network

At least 13 per cent of research abstracts published in 2024 could have taken help from a large language model, as they included more of ‘style’ words seen to be favoured by these AI systems, suggests an analysis of more than 15 million biomedical papers published from 2010 to 2024.

Powered by artificial intelligence, large language models are trained on vast amounts of text and can, therefore, respond to human requests in the natural language.

Researchers from the University of Tübingen, Germany, said the AI models have caused a drastic shift in the vocabulary used in academic writing, with speculation about their influence in scientific writing being common.

The study, published in the journal Science, revealed the emergence of large language models has sparked an increase in the usage of certain “stylistic words”, including ‘delves’, ‘showcasing’, ‘underscores’, ‘potential’, ‘findings’ and ‘critical’.

The authors explained that the shift in words used during 2023-2024 were not “content-related nouns”, rather style-affecting verbs and adjectives that large language models prefer.

For the analysis, the researchers used a public health approach, common during the COVID-19 pandemic, for estimating excess deaths. The method involves comparing deaths during the pandemic with those before to assess the impact of COVID-19 on death rates.

The approach modified for this analysis was termed as an “excess word” framework by researchers.

The findings show an “unprecedented impact” of AI models on scientific writing in biomedical research, “surpassing the effect of major world events such as the COVID-19 pandemic”.

“We study vocabulary changes in more than 15 million biomedical abstracts from 2010 to 2024 indexed by PubMed and show how the appearance of (large language models) led to an abrupt increase in the frequency of certain style words,” the authors wrote.

PubMed is a search engine providing access to biomedical and life sciences literature published from around the world.

“This excess word analysis suggests that at least 13.5 per cent of 2024 abstracts were processed with (large language models),” the team wrote.

The figure was found to differ across disciplines, countries and journals, hitting 40 per cent in some cases, they said.

In computational fields of biomedical research, about 20 per cent of the abstracts involved the use of large language models, which the researchers said could be due to computer science researchers being more familiar with and willing to adopt the technology.

In non-English speaking countries, the AI systems can help authors with editing English texts, which could justify their extensive use, the authors said.

However, they added that factors such as publication timelines—which are shorter in computational fields, thereby enabling an earlier detection of AI use in these journals—would need to be looked at.

Therefore, the study’s results may be re-evaluated after a couple of publication cycles in all fields and journals for which the methods used here can help, the team said.