Proposal: Betting Reviewed vs. Peer Reviewed

A Comparative Analysis of Prediction Markets and Traditional Surveys in Forecasting Biomedical Research Outcomes

1. Overview

This project tests whether incentive-aligned forecasting mechanisms outperform traditional survey-based peer review in predicting the outcomes of preregistered biomedical studies. Specifically, we compare (1) non-incentivized surveys, (2) fixed-odds betting, and (3) prediction markets on questions such as study completion timelines, hypothesis confirmation, and publication tier. The study will be conducted by independent researchers (BetBreakingNews) in collaboration with Texas State University’s Center for Analytics and Data Science, using existing preregistered studies hosted on ResearchHub. Ethics approval will be sought or confirmed prior to launch. The project is feasible within 6 months. All data will be shared openly in accordance with ResearchHub’s policies.

Note to all that donate to this project:

40% of all fees collected through the betting and prediction market platforms will be given to the laboratories running the research that we are tracking with our markets.
These markets will be using the RSC ResearchHub token as the form of currency.
Donors who reach the following tiers will receive an account for the betting and prediction market platforms with the following perks:
- Donations totalling between $100 and 499 will 20% of their donation as RSC credit for the betting/prediction platforms
- Donations totalling between $500 and 999 will 25% of their donation as RSC credit for the betting/prediction platforms
- Donations at $1000 or more will 30% of their donation as RSC credit for the betting/prediction platforms

[Moonshot Goal]: If we raise more than $85k, every dollar beyond that will go into a dedicated fund to support prediction market/betting intelligence research within the ResearchHub ecosystem.

2. Introduction

Peer review is a cornerstone of modern science, yet it is well known to be noisy, slow, and often poor at forecasting real-world research outcomes such as reproducibility, timelines, and impact. Prior work suggests that prediction markets and incentive-aligned forecasting systems can outperform unaided expert judgment in domains ranging from geopolitics to economics. However, systematic tests of these mechanisms in biomedical research evaluation remain limited.

This study asks: Do market-based mechanisms produce more accurate and responsive forecasts about preregistered biomedical studies than standard survey-based reviews? We focus on concrete, observable outcomes: time-to-completion milestones, hypothesis confirmation, and eventual publication tier.

Research Questions:

Does the introduction of a financial stake (RSC token) increase the accuracy of research milestone forecasting?
Does a dynamic prediction market adapt more quickly to "lab updates" than a static betting book?

Hypotheses:

H1: Both betting-based conditions (fixed-odds and prediction markets) will produce more accurate forecasts than non-incentivized surveys.
H2: Prediction markets will update more rapidly and sensitively in response to new information than fixed-odds betting.
H3: Overall forecast accuracy between fixed-odds betting and prediction markets will be similar.

Impact Statement: Regardless of outcome, this study will provide the first direct, preregistered comparison of peer-review-like surveys versus market mechanisms for evaluating biomedical research trajectories. Positive results would motivate new AI-assisted, market-informed tools for funding, publication triage, and project management; negative results would clarify the limits of incentive mechanisms in scientific forecasting.

3. Methods

Participants/Sample

Target: 200–500 participants (over the age of 18) from the ResearchHub and academic community (verified via ORCID).
Recruitment: Targeted outreach on ResearchHub, Twitter/X (Science), and Texas State University networks.

Materials and Procedures

We will evaluate 10–15 active biomedical pre-registrations on ResearchHub. Participants will be randomly assigned to one of three arms:

Group A (Control): SurveyMonkey/Google Surveys (No stakes).
Group B (Book): A "Betting Book" platform that we develop where odds are set and positions are locked once placed.
Group C (Market): A Prediction Market platform where we develop where users buy/sell shares in outcomes (e.g., "Will this reach a Tier 1 Journal?").

All conditions will forecast the same set of questions for the same studies. Outcomes will be scored against real-world results as studies progress and conclude.

Planned Analyses

Primary outcome: forecast accuracy, measured via Brier scores and absolute error for probabilistic predictions, and classification accuracy for categorical outcomes.

Test H1: IF mean accuracy in betting conditions > survey condition, THEN conclude incentive alignment improves forecasting.
Test H2: IF prediction markets show faster and larger belief updates following new information releases, THEN conclude higher responsiveness.
Test H3: IF no statistically significant difference in final accuracy between the two betting conditions, THEN conclude comparable accuracy.

Secondary analyses will examine participation patterns, liquidity effects, and expertise subgroups.

Ethics and Data Management

Ethics approval will be obtained or confirmed prior to data collection. No sensitive personal data will be collected beyond basic demographics and consent. All anonymized data, analysis code, and instruments will be shared in an open-access repository in line with ResearchHub requirements.

4. Pilot Data

All data collected in this study will be released to the public with the publication of results.

5. Budget

Total Requested: $25,000 (Initial RSC Grant: 5,000; Community Crowdfund: 20,000)

Item	Estimated Cost	Description
Platform Development	$11,000	Building the betting/market UI integrated with RSC.
Liquidity Seed	$9,000	Initial RSC pool to ensure market depth.
Admin/Data Analysis	$3,000	Statistical support from Texas State CADS.
Marketing/Recruitment	$2,000	Targeted ads and ResearchHub "Boosts".

6. References

Dreber, A. et al. (2015). Using prediction markets to estimate the reproducibility of scientific research. PNAS.

Hanson, R. (2003). Combinatorial information market design. Information Systems Frontiers.

Proposal: Betting Reviewed vs. Peer Reviewed

Proposal: Betting Reviewed vs. Peer Reviewed

Funding Progress

Funders

Related Funding

Request for Proposals - AI Tools & AI Agents for Biomedical Research

Nonprofit Recipient

Topics

DOI