Tracking claim changes from preprint to publication across biomedical studies using large language models

Authors

Hao Yin¹, Ruslan Rust^2,3

Affiliations

¹ Robarts Research Institute, Schulich School of Medicine and Dentistry, Western University, London, Ontario, N6A 5C1, Canada

² Department of Physiology and Neuroscience, University of Southern California, Los Angeles, CA 90033, USA

³ Zilkha Neurogenetic Institute, Keck School of Medicine, University of Southern California, Los Angeles, CA 90033, USA

Correspondence

Ruslan Rust, Ph.D.
Assistant Professor
The Zilkha Neurogenetic Institute,
Department of Physiology and Neuroscience
Keck School of Medicine of the University of Southern California
1501 San Pablo Street
Los Angeles, CA 90033
email: rrust@usc.edu

ORCID
Hao Yin: 0000-0002-0018-3228
Ruslan Rust: 0000-0003-3376-3453

1. Overview

Peer review is the primary quality-control mechanism in science, yet its specific impact on the claims authors make remains poorly quantified at scale. Comparing the preprints and their peer-reviewed versions provides a value window to evaluate this. Specifically, we would like to examine whether and how scientific claims are changed between preprints and peer-reviewed publications, which is enabled by the extensive use of preprint servers (bioRxiv, medRxiv, arXiv, et al) and the advancement of large language models (LLMs). In this pre-registered study we will test how the primary and secondary claims of bioRxiv abstracts change between preprint deposition and journal publication, using Claude Sonnet and Haku for structured claim extraction. We will analyze 72,644 matched preprint-publication pairs across all bioRxiv categories (2018-2025), classifying each pair by 4 claim-level annotations: (1) content change (unchanged, minor, major), (2) hedging shift (more cautious, more confident, unchanged), (3) claim type (mechanism, association, descriptive, method, therapeutic, null result), and (4) study field, and we will compare retraction rates between preprinted and never-preprinted papers. A feasibility test run of the full pipeline confirms the LLM approach is technically feasible, which we report as Pilot Data in this preregistered report. This study requires no ethics approval, as only publicly available data are used. Computational costs are covered by existing lab funds (~$500 for LLM API, ~$2,000 for human validation). Timeline: 2-3 weeks. All extracted claim data will be shared openly on GitHub and Zenodo.

2. Introduction

The preprint ecosystem has fundamentally changed scientific communication. BioRxiv hosts over 300,000 preprints, the majority of which eventually appear in peer-reviewed journals¹. The rapid growth has made preprints an important early record of scientific work, but it also raises a critical question – how much do the scientific claims change with peer review process. Emerging evidence sheds lights but remains imcomplete. Brierley et al. (2022) found that 7.2% of non-COVID and 17.2% of COVID-related preprints had discrete conclusion changes, but relied on manual annotation of a limited sample². Nicholson et al. (2022) tracked linguistic shifts more broadly and found most changes relate to formatting rather than substantive change³. Similarly, Nelson et al. (2022) demonstrated remarkable stability of quantitative point estimates (r = 0.99 between preprint and published values)⁴. Furthermore, Peng et al. (2024) demonstrated that the degree of manuscript revision correlates with later academic impact, with introduction sections showing the most change⁵.

Critically, no study has systematically categorized both the type and direction of claim changes across disciplines at scale. Several important questions remain unanswered: Are mechanistic claims revised differently from therapeutic one? Do high-impact journals drive more substantial changes? How long the revision process takes for different types of modifications? Our study addresses these gaps using LLM-based structured claim extraction on the largest sample to date.

Hypotheses

Peer review does not substantially change the central claims of biomedical preprints. We predict that the primary claim is unchanged or only minorly revised in the large majority of preprint-publication pairs, with a major change occurring in only a small minority, and that this pattern is consistent across bioRxiv study fields.
When claims are revised, the revision is more often conservative than inflationary. We predict that words shifts toward caution (weakening) more often than toward confidence (strengthening), yielding a ratio of strengthened to weakened claims below one, particularly for mechanistic and therapeutic claims.
Preprinting is not associated with greater unreliability. We predict that papers posted as preprints are retracted no more often than comparable papers that were never posted as preprints, with a retraction rate ratio at or below one.

Impact statement. This study will produce the largest claim-level dataset of preprint-to-publication change to date. If most claims stable throughout peer review process and preprints are not retracted more often, the findings would support the use of preprints as a reliable early record of biomedical research. Conversely, if specific claim types change systematically or preprints are retracted more often, this adds cautions to preprint credibility and informs review policy. Broadly speaking, the validated LLM framework will provide a scalable tool for ongoing monitoring of the evolution of preprints and peer reviews.

3. Methods

Sample. All bioRxiv preprints (2018-2025) matched by DOI to a linked peer-reviewed publication. Assembled corpus: 72,644 matched pairs across all bioRxiv categories and thousands of journals. Inclusion: both abstracts available in English, >50 words each; the first preprint version is used for duplicates. For H3, the retraction status of these preprinted papers is compared with that of journal articles from the same period and fields that were never posted as a preprint.

Sample size justification. With N > 72,000, the claim-change analyses have >99% power to detect 5-percentage-point differences across fields (two-proportion z-test, Bonferroni-corrected alpha = 0.005). The LLM-human agreement benchmark uses a stratified validation subset of 120 pairs (95% CI of approximately +/- 0.09 for Cohen's kappa). Retractions are rare events, so the H3 comparison is powered only to detect large differences in retraction rate and is interpreted with corresponding caution.

Claim extraction. For each abstract, a large language model (Claude Sonnet 4.6) extracts: primary claim (one sentence), claim type (mechanism, association, method, descriptive, therapeutic, null_result), up to 2 secondary claims with types, and hedging level. A second prompt compares preprint and published extractions and classifies content change (unchanged, minor, major) and hedging shift (more cautious, unchanged, more confident). The model version and prompts are fixed in advance and reported following current LLM reporting standards.

Planned analyses. H1: estimate the proportion of pairs that are unchanged, minorly, and majorly revised with 95% confidence intervals, and the major-revision rate by field; H1 is supported if major revision occurs in a small minority (pre-specified as <20%) overall and across fields. H2: Wilcoxon signed-rank test on paired hedging scores and a sign test that weakened exceeds strengthened (strengthened-to-weakened ratio < 1), with a logistic contrast of therapeutic/mechanistic vs. descriptive/associative claims. H3: compare retraction rates of preprinted vs never-preprinted papers with a Poisson rate ratio and 95% CI; H3 is supported if the rate ratio is at or below one. Exploratory: association of revision magnitude with time-to-publication and journal impact factor (linear regression with restricted cubic splines).

Contingent decisions. IF overall change rate < 10%, THEN we add word-level text metrics (Levenshtein, semantic similarity). IF any field has < 1,500 papers, THEN it is merged with related categories or excluded from field-level analysis. IF LLM-human kappa < 0.40, THEN prompts are revised and re-validated before confirmatory extraction. IF retraction events number fewer than ten per group, THEN the H3 comparison is reported descriptively rather than inferentially.

Ethics and data sharing. The study involves no human subjects and use publicly available data only. Code will be shared on GitHub; extracted claims will be deposited on Zenodo under a CC-BY 4.0 license.

4. Pilot Data

To confirm feasibility, we ran the complete labeling pipeline on a pilot sample of 1,000 preprint-publication pairs drawn at random from the assembled corpus and stratified across bioRxiv fields. This test run shows that the workflow can parse each abstract into structured claims and assign content-change and hedging labels, and scale to the full corpus. No confirmatory results are reported here; the pre-registered analyses will be run on the full corpus only after this plan is registered.

Data collection feasibility. All 72,644 candidate pairs were retrieved via the bioRxiv API and matched by DOI to their published versions, confirming that corpus assembly is feasible at scale. The pilot sample of 1,000 pairs was drawn from this corpus, stratified across bioRxiv fields, with English-language abstracts available for both preprint and peer-reviewed versions.

Claim extraction feasibility. The model parsed every abstract in the pilot set into one primary claim and two secondary claims, and it returned a valid content-change and hedging label for each pair, with zero parsing failures. On a stratified validation subset, model agreement with two independent human domain experts was similar to agreement between the human experts (Cohen's kappa >0.60), Three replicate model runs agreed at kappa >0.75, confirming the reproducibility. These results establish that claims can be labeled reliably at scale; the distribution of label categories is not interpreted here and will be estimated only in the confirmatory analysis.

5. Budget

Item	Cost	Notes
Claude API (claim extraction)	$900	~300k calls, Haiku/Sonnet
Human rater compensation	$0 (covered elsewhere)	2 raters, 300 papers
Open access publication	$0 (covered elsewhere)	-
Total	$900	-

6. References

Abdill, R. J. & Blekhman, R. Tracking the popularity and outcomes of all bioRxiv preprints. eLife 8, e45133 (2019).
Brierley, L. et al. Tracking changes between preprint posting and journal publication during a pandemic. PLoS Biol. 20, e3001285 (2022).
Nicholson, D. N. et al. Examining linguistic shifts between preprints and publications. PLoS Biol. 20, e3001470 (2022).
Nelson, L. et al. Robustness of evidence reported in preprints during peer review. Lancet Glob. Health 10, e1684-e1687 (2022).
Peng, W. et al. Revision and academic impact: a case study of bioRxiv preprint papers. J. Informetr. 18, 101484 (2024).

Figure 1: Graphical abstract

Tracking claim changes from preprint to publication across biomedical studies using large language models

Tracking claim changes from preprint to publication across biomedical studies using large language models

Funding Progress

Funders

Nonprofit Recipient

Topics