This is adapted from our recent paper in F1000 Research, entitled “A multi-disciplinary perspective on emergent and future innovations in peer review.” Due to its rather monstrous length, I’ll be posting chunks of the text here in sequence over the next few weeks to help disseminate it in more easily digestible bites. Enjoy! This section describes some of the recent studies investigating peer review. Next time, the role and purpose of modern peer review!
Evidence from studies of peer review.
Several empirical studies on peer review have been reported in the past few decades, mostly at the journal- or population-level. These studies typically use several different approaches to gather evidence on the functionality of peer review. Some, such as Bornmann & Daniel (2010b); Daniel (1993); Zuckerman & Merton (1971), used access to journal editorial archives to calculate acceptances, assess inter-reviewer agreement, and compare acceptance rates to various article, topic, and author features. Others interviewed or surveyed authors, reviewers, and editors to assess attitudes and behaviours, while others conducted randomized controlled trials to assess aspects of peer review bias (Justice et al., 1998; Overbeke, 1999). A systematic review of these studies concluded that evidence supporting the effectiveness of peer review training initiatives was inconclusive (Galipeau et al., 2015), and that major knowledge gaps existed in our application of peer review as a method to ensure high quality of scientific research outputs.
In spite of such studies, there appears to be a widening gulf between the rate of innovation and the availability of quantitative, empirical research regarding the utility and validity of modern peer review systems (Squazzoni et al., 2017a; Squazzoni et al., 2017b). This should be deeply concerning given the significance that has been attached to peer review as a form of community moderation in scholarly research. Indeed, very few journals appear to be committed to objectively assess their effectiveness during peer review (Lee & Moher, 2017). The consequence of this is that much remains unknown about the “black box” of peer review, as it is sometimes called (Smith, 2006). The optimal designs for understanding and assessing the effectiveness of peer review, and therefore improving it, remain poorly understood, as the data required to do so are often not available (Bruce et al., 2016; Galipeau et al., 2015). This also makes it very hard to measure and assess the quality, standard, and consistency of peer review not only between articles and journals, but also on a system-wide scale in the scholarly literature. Research into such aspects of peer review is quite time-consuming and intensive, particularly when investigating traits such as validity, and often criteria for assessing these are based on post-hoc measures such as citation frequency.
Despite the criticisms levied at the implementation of peer review, it remains clear that the ideal of it still plays a fundamental role in scholarly communication (Goodman et al., 1994; Mulligan et al., 2013; Pierie et al., 1996; Ware, 2008) and retains a high level of respect from the research community (Bedeian, 2003; Gibson et al., 2008; Greaves et al., 2006; Mulligan et al., 2013). One primary reason why peer review has persisted is that it remains a unique way of assigning credit to authors and differentiating research publications from other types of literature, including blogs, media articles, and books. This perception, combined with a general lack of awareness or appreciation of the historic evolution of peer review, research examining its potential flaws, and the conflation of the process with the ideology, has sustained its near-ubiquitous usage and continued proliferation in academia. There remains a widely-held perception that peer review is a singular and static process, and thus its wide acceptance as a social norm. It is difficult to move away from a process that has now become so deeply embedded within global research institutes. The consequence of this is that validation offered through peer review remains one of the essential pillars of trust in scholarly communication, irrespective of any potential flaws (Haider & Åström, 2017).
In the following section, we summarize the ebb and flow of the debate around the various and complex aspects of conventional peer review. In particular, we highlight how innovative systems are attempting to resolve some of the major issues associated with traditional models, explore how new platforms could improve the process in the future, and consider what this means for the identity, role, and purpose of peer review within diverse research communities. The aim of this discussion is not to undermine any specific model of peer review in a quest for systemic upheaval, or to advocate any particular alternative model. Rather, we acknowledge that the idea of peer review is critical for research and advancing our knowledge, and as such we provide a foundation for future exploration and creativity in improving an essential component of scholarly communication.
Tennant JP, Dugan JM, Graziotin D et al. A multi-disciplinary perspective on emergent and future innovations in peer review [version 2; referees: 2 approved]. F1000Research 2017, 6:1151 (doi: 10.12688/f1000research.12037.2)