Five years of submissions to Science and Science Advances: Vermont Complex Systems looks behind-the-scenes
What happens when a paper is submitted to Science? VCSI core faculty Sam Zhang opens the black box
A project by:
Elite general science journals shape scientific discourse, public policy, and scientific careers. Publishing a manuscript in one of these journals can alter the direction not just of an individual career, but of an entire scientific field. Therefore, scientists have a vested interest in understanding the editorial processes shaping outcomes at these journals. However, scientific journals also have a long tradition of reviewer confidentiality, making it difficult to allow the broader scientific community a closer look.
Among the most prestigious and impactful journals are Science and Science Advances, two general science journals published by the American Association for the Advancement of Science (AAAS). With collaborators, director of VCSI’s Science and Humanity Lab Sam Zhang worked with AAAS to de-identify their editorial data, and publish a dataset of the editorial outcomes of 110,303 manuscripts submitted between 2015-2020 for the two journals [eventual citation to paper here?]. This dataset allows the scientific community to understand elite peer review as a complex system.
The editorial process at elite journals typically operates like a black box — papers go in, decisions come out, but quantitative data about what happens inside is largely hidden from view.
VCSI faculty collaborated with the American Association for the Advancement of Science to open the box — and allow the entire scientific community to trace what happens to every paper submitted to Science and Science Advances.
Between 2015 and 2019, there were 68,047 manuscripts submitted to science. Each square represents one paper. By the time they go through the review process, only 6% are accepted to the journal.
The newly released AAAS data allows the scientific community to quantify the editorial dynamics of submissions to Science (and Science Advances) over a five year period. Most submissions are rejected, and there are multiple stages where rejection can occur. A successful submission must clear several hurdles by multiple sets of evaluators.
First, the editor rejects certain unsuitable submissions immediately. The submissions not immediately rejected are typically routed to a small group of several hundred external experts, called the Board of Reviewing Editors (BoRE), who quickly evaluate the manuscript and provide a numeric score and suggested reviewers. BoRE feedback and scores is not made available to authors, but it is present in the new dataset.
Based on BoRE feedback, the editor rejects another set of submissions. The remaining manuscripts are forwarded to peer reviewers. The editor makes the decision whether to proceed to subsequent rounds of review, acceptance, or rejection, based on the comments from the peer reviewers.
68,047 papers were submitted to Science between 2010 and 2019. Each square represents a paper flowing through the review pipeline.
Every submission first enters the Board of Reviewing Editors — a brief editorial review. About 30% (20,755) are rejected here without further review.
Papers that clear the Board of Reviewing Editors (BoRE) may be sent to peer review. Only 1 in 4 BoRE-approved papers (11,845) reaches this stage — the rest are rejected by editors.
Of papers reaching peer review, 4,181 are accepted — just 6% of all submissions. The other 7,664 are rejected after review.
This dataset provides the community an unprecedented opportunity to glimpse into the inner workings of peer review, and to quantify the correlates of outcome disparities. The team found higher acceptance rates among larger teams whose corresponding authors worked in the US, Canada, or Europe, and those at prestigious institutions, as well as a small gender disparity (Fig. 1). However, these results are not causal, and there can be unmeasured attributes that explain these disparities.
Randomized controlled experiments at the journals, such as through double-blinding, can help measure the causal effect of bias compared to other unmeasured correlates such as the manuscript’s novelty or quality. The release of this data by the AAAS serves as a welcome example for the scientific community, and the collaboration with academic institutions provides a template for how other academic journals can demonstrate their public commitment to transparency.