This is adapted from our recent paper in F1000 Research, entitled “A multi-disciplinary perspective on emergent and future innovations in peer review.” Due to its rather monstrous length, I’ll be posting chunks of the text here in sequence over the next few weeks to help disseminate it in more easily digestible bites. Enjoy!
This section describes the debate around whether peer review reports should be published. Next time we’ll have a look at the issues surrounding anonymity in peer review. Previous sections:
- Peer Review: An Introduction.
- Peer Review: An Early History
- Peer Review: The Modern Revolution
- Peer Review: Recent Studies
- Peer Review: Modern Role and Purpose
- Peer Review: Criticisms of the Conventional System
- Peer Review: Modern Trends and Traits
- Peer Review: Development of Open Peer Review
- Peer Review: Giving Credit to Referees
Publishing peer review reports
The rationale behind publishing referee reports lies in providing increased context and transparency to the peer review process, and can occur irrespective of whether or not the reviewers reveal their identities. Often, valuable insights are shared in reviews that would otherwise remain hidden if not published. By publishing reports, peer review has the potential to become a supportive and collaborative process that is viewed more as an ongoing dialogue between groups of scientists to progressively assess the quality of research. Furthermore, the reviews themselves are opened up for analysis and inspection, including how authors respond to reviews, adding an additional layer of quality control and a means for accountability and verification.
There are additional educational benefits to publishing peer reviews, such as training purposes or for journal clubs. Given the inconclusive evidence regarding the training of referees (Galipeau et al., 2015; Jefferson et al., 2007), such practices might be further useful in highlighting our knowledge and skills gaps. At the present, some publisher policies are extremely vague about the re-use rights and ownership of peer review reports (Schiermeier, 2017). The Peer Review Evaluation (PRE) service (www.pre-val.org) was designed to breathe some transparency into peer review, and provide information about the peer review itself without exposing the reports (e.g., mode of peer review, number of referees, rounds of review). While it describes itself as a service to identify fraud and maintain the integrity of peer review, it remains unclear whether it has achieved these objectives in light of the ongoing criticisms of the conventional process.
In a study of two journals, one where reports were not published and another where they were, Bornmann et al. (2012)found that publicized comments were much longer by comparison. Furthermore, there was an increased chance that they would result in a constructive dialogue between the author, reviewers, and wider community, and might therefore be better for improving the content of a manuscript. On the other hand, unpublished reviews tended to have more of a selective function to determine whether a manuscript is appropriate for a particular journal (i.e., focusing on the editorial process). Therefore, depending on the journal, different types of peer review could be better suited to perform different functions, and therefore optimized in that direction. Transparency of the peer review process can also be used as an indicator for peer review quality, thereby potentially enabling the tool to predict quality in new journals in which the peer review model is known, if desired (Godlee, 2002; Morrison, 2006; Wicherts, 2016). Journals with higher transparency ratings were less likely to accept flawed papers and showed a higher impact as measured by Google Scholar’s h5-index (Wicherts, 2016).
Assessments of research articles can never be evidence-based without the verification enabled by publication of referee reports. However, they are still almost ubiquitously regarded as having an authoritative, and uniform, stamp of quality. The issue here is that the attainment of peer reviewed status will always be based on an undefined, and only ever relative, quality threshold due to the opacity of the process. This is in itself quite an unscientific practice, and instead, researchers rely almost entirely on heuristics and trust for a concealed process and the intrinsic reputation of the journal, rather than anything legitimate. This can ultimately result in what is termed the “Fallacy of Misplaced Finality”, described by Kelty et al. (2008), as the assumption that research has a single, final form, to which everyone applies different criteria of quality.
Publishing peer review reports appears to have little or no impact on the overall process but may encourage more civility from referees. In a small survey, Nicholson & Alperin (2016) found that approximately 75% of survey respondents (n=79) perceived that public peer review would change the tone or content of the reviews, and 80% of responses indicated that performing peer reviews that would be eventually be publicized would not require a significantly higher amount of work. However, the responses also indicated that incentives are needed for referees to engage in this form of peer review. This includes recognition by performance review or tenure committees (27%), peers publishing their reviews (26%), being paid in some way such as with an honorarium or waived APC (24%), and getting positive feedback on reviews from journal editors (16%). Only 3% (one response) indicated that nothing could motivate them to participate in an open peer review of this kind.
Leek et al. (2011) showed that when referees’ comments were made public, significantly more cooperative interactions were formed, while the risk of incorrect comments decreased, suggesting that prior knowledge of publication encourages referees to be more constructive and careful with their reviews. Moreover, referees and authors who participated in cooperative interactions had a reviewing accuracy rate that was 11% higher. On the other hand, the possibility of publishing the reviews online has also been associated with a high decline rate among potential peer reviewers, and an increase in the amount of time taken to write a review, but with a variable effect on review quality (Almquist et al., 2017; van Rooyen et al., 2010). This suggests that the barriers to publishing review reports are inherently social, rather than technical.
When BioMed Central launched in 2000, it quickly recognized the value in including both the reviewers’ names and the peer review history (pre-publication) alongside published manuscripts in their medical journals in order to increase the quality and value of the process. Since then, further reflections on OPR (Godlee, 2002) led to the adoption of a variety of new models. For example, the Frontiers series now publishes all referee names alongside articles, EMBO journals publish a review process file with the articles, with referees remaining anonymous but editors being named, and PLOS added public commenting features to articles they published in 2009. More recently launched journals such as PeerJ have a system where both the reviews and the names of the referees can optionally be made public, and journals such as Nature Communications and the European Journal of Neuroscience have also started to adopt this method.
Unresolved issues with posting review reports include whether or not it should be conducted for ultimately unpublished manuscripts, and the impact of author identification or anonymity alongside their reports. Furthermore, the actual readership and usage of published reports remains ambiguous in a world where researchers are typically already inundated with published articles to read. The benefits of publicizing reports might not be seen until further down the line from the initial publication and, therefore, their immediate value might be difficult to convey and measure in current research environments. Finally, different populations of reviewers with different cultural norms and identities will undoubtedly have varying perspectives on this issue, and it is unlikely that any single policy or solution to posting referee reports will ever be widely adopted. Further investigation of the link between making reviews public and the impact this has on their quality would be a fruitful area of research to potentially encourage increased adoption of this practice.
Tennant JP, Dugan JM, Graziotin D et al. A multi-disciplinary perspective on emergent and future innovations in peer review [version 3; referees: 2 approved]. F1000Research 2017, 6:1151 (doi: 10.12688/f1000research.12037.3)