Identify whether each of the following is a reason for peer review or a problem with peer review.

1. T. I. R. Institute. 2017 R&D trends forecast: results from the Industrial Research Institute’s annual survey. Res Technol Manag. 2017;60:18–25.

   Google Scholar 

2. R. Johnson, A. Watkinson, M. Mabe, The STM Report: an overview of scientific and scholarly publishing. International Association of Scientific, Technical and Medical Publishers (2018).

3. Rennie D. Editorial peer review: its development and rationale. Peer Rev Health Sci. 2003;2:1–13.

   Google Scholar 

4. C. Neylon, Arenas of productive conflict: Universities, peer review, conflict and knowledge (2018) (available at https://hcommons.org/deposits/item/hc:22483/).

5. J. P. Tennant, B. Penders, T. Ross-Hellauer, A. Marušić, F. Squazzoni, A. W. Mackay, C. R. Madan, D. M. Shaw, S. Alam, B. Mehmani, Boon, bias or bane? The potential influence of reviewer recommendations on editorial decision-making (2019).

6. Moher D, Galipeau J, Alam S, Barbour V, Bartolomeos K, Baskin P, Bell-Syer S, Cobey KD, Chan L, Clark J, Deeks J, Flanagin A, Garner P, Glenny A-M, Groves T, Gurusamy K, Habibzadeh F, Jewell-Thomas S, Kelsall D, Lapeña JF, MacLehose H, Marusic A, McKenzie JE, Shah J, Shamseer L, Straus S, Tugwell P, Wager E, Winker M, Zhaori G. Core competencies for scientific editors of biomedical journals: consensus statement. BMC Med. 2017;15:167.

   Google Scholar 

7. Overbeke J, Wager E. 3: The state of evidence: what we know and what we don’t know about journal peer review. JAMA. 2011;272:79–174.

   Google Scholar 

8. Malički M, von Elm E, Marušić A. Study design, publication outcome, and funding of research presented at International Congresses on Peer Review and Biomedical Publication. JAMA. 2014;311:1065–7.

   Google Scholar 

9. Dondio P, Casnici N, Grimaldo F, Gilbert N, Squazzoni F. The “invisible hand” of peer review: the implications of author-referee networks on peer review in a scholarly journal. J Inform. 2019;13:708–16.

   Google Scholar 

10. Grimaldo F, Marušić A, Squazzoni F. Fragments of peer review: a quantitative analysis of the literature (1969-2015). PLOS ONE. 2018;13:e0193148.

    Google Scholar 

11. Batagelj V, Ferligoj A, Squazzoni F. The emergence of a field: a network analysis of research on peer review. Scientometrics. 2017;113:503–32.

    Google Scholar 

12. Ross-Hellauer T. What is open peer review? A systematic review. F1000Research. 2017;6:588.

    Google Scholar 

13. Allen H, Boxer E, Cury A, Gaston T, Graf C, Hogan B, Loh S, Wakley H, Willis M. What does better peer review look like? Definitions, essential areas, and recommendations for better practice. Open Sci Framework. 2018. https://doi.org/10.17605/OSF.IO/4MFK2.

14. S. Parks, S. Gunashekar, Tracking Global Trends in Open Peer Review (2017; https://www.rand.org/blog/2017/10/tracking-global-trends-in-open-peer-review.html).

15. Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006;99:178–82.

    Google Scholar 

16. Groves T. Is open peer review the fairest system? Yes. BMJ. 2010;341:c6424.

    Google Scholar 

17. Khan K. Is open peer review the fairest system? No. BMJ. 2010;341:c6425.

    Google Scholar 

18. Smith R. Peer review: reform or revolution? Time to open up the black box of peer review. BMJ. 1997;315:759–60.

    Google Scholar 

19. Relman AS. Peer review in scientific journals--what good is it? West J Med. 1990;153:520–2.

    Google Scholar 

20. Kassirer JP, Campion EW. Peer review: crude and understudied, but indispensable. JAMA. 1994;272:96–7.

    Google Scholar 

21. Wessely S. What do we know about peer review? Psychol Med. 1996;26:883–6.

    Google Scholar 

22. Ross-Hellauer T, Deppe A, Schmidt B. Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers. PLOS ONE. 2017;12:e0189311.

    Google Scholar 

23. Tennant JP, Dugan JM, Graziotin D, Jacques DC, Waldner F, Mietchen D, Elkhatib Y, Collister LB, Pikas CK, Crick T, Masuzzo P, Caravaggi A, Berg DR, Niemeyer KE, Ross-Hellauer T, Mannheimer S, Rigling L, Katz DS, Tzovaras BG, Pacheco-Mendoza J, Fatima N, Poblet M, Isaakidis M, Irawan DE, Renaut S, Madan CR, Matthias L, Kjær JN, O’Donnell DP, Neylon C, Kearns S, Selvaraju M, Colomb J. A multi-disciplinary perspective on emergent and future innovations in peer review. F1000Research. 2017;6:1151.

    Google Scholar 

24. Kaplan D. How to fix peer review: separating its two functions—improving manuscripts and judging their scientific merit—would help. J Child Fam Stud. 2005;14:321–3.

    Google Scholar 

25. Hunter J. Post-publication peer review: opening up scientific conversation. Front. Comput. Neurosci. 2012;6. https://doi.org/10.3389/fncom.2012.00063.

26. Csiszar A. Peer review: troubled from the start. Nat News. 2016;532:306.

    Google Scholar 

27. Baldwin M. Credibility, peer review, and Nature, 1945–1990. Notes Rec. 2015;69:337–52.

    Google Scholar 

28. Moxham N, Fyfe A. The Royal Society And the prehistory of peer review, 1665–1965. Historical J. 2017:1–27.

29. A. Fyfe, K. Coate, S. Curry, S. Lawson, N. Moxham, C. M. Røstvik, Untangling Academic Publishing. A history of the relationship between commercial interests, academic prestige and the circulation of research., 26 (2017).

30. R. Wijesinha-Bettoni, K. Shankar, A. Marusic, F. Grimaldo, M. Seeber, B. Edmonds, C. Franzoni, F. Squazzoni, Reviewing the review process: new frontiers of peer review. Editorial Board, 82 (2016).

31. Squazzoni F, Brezis E, Marušić A. Scientometrics of peer review. Scientometrics. 2017;113:501–2.

    Google Scholar 

32. Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, du Sert NP, Simonsohn U, Wagenmakers E-J, Ware JJ, Ioannidis JPA. A manifesto for reproducible science. Nat Human Behav. 2017;1:0021.

    Google Scholar 

33. O. S. Collaboration. Estimating the reproducibility of psychological science. Science. 2015;349:aac4716.

    Google Scholar 

34. Crick T, Hall B, Ishtiaq S. Reproducibility in research: systems, infrastructure, culture. J Open Res Software. 2017;5:32.

    Google Scholar 

35. ter Riet G, Storosum BWC, Zwinderman AH. What is reproducibility? F1000Res. 8:36, 2019.

36. L. A. Barba, Terminologies for reproducible research. arXiv:1802.03311 [cs] (2018) (available at http://arxiv.org/abs/1802.03311).

37. Bravo G, Grimaldo F, López-Iñesta E, Mehmani B, Squazzoni F. The effect of publishing peer review reports on referee behavior in five scholarly journals. Nat Commun. 2019;10:322.

    Google Scholar 

38. Squazzoni F, Grimaldo F, Marušić A. Publishing: Journals could share peer-review data. Nature. 2017. https://doi.org/10.1038/546352a.

39. S. Pranić, B. Mehmani, S. Marušić, M. Malički, A. Marušić, in New Frontiers of Peer Review (PEERE), European Cooperation in Science and Technology (COST) (2017).

40. Allen H, Cury A, Gaston T, Graf C, Wakley H, Willis M. What does better peer review look like? Underlying principles and recommendations for better practice. Learned Publishing. 2019;32:163–75.

    Google Scholar 

41. J. C. Bailar III, K. Patterson, Journal peer review: the need for a research agenda (Mass Medical Soc, 1985).

42. Lee CJ, Moher D. Promote scientific integrity via journal peer review data. Science. 2017;357:256–7.

    Google Scholar 

43. van Rooyen S, Delamothe T, Evans SJW. Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ. 2010;341:c5729.

    Google Scholar 

44. Polka JK, Kiley R, Konforti B, Stern B, Vale RD. Publish peer reviews. Nature. 2018;560:545.

    Google Scholar 

45. Hope AA, Munro CL. Criticism and judgment: a critical look at scientific peer review. Am J Crit Care. 2019;28:242–5.

    Google Scholar 

46. B.-C. Bjórk, Acceptance rates of scholarly peer-reviewed journals: a literature survey. El Profesional de la Información. 28 (2019), doi:10/gf6zzk.

47. Sugimoto CR, Larivière V, Ni C, Cronin B. Journal acceptance rates: a cross-disciplinary analysis of variability and relationships with journal measures. J Inform. 2013;7:897–906.

    Google Scholar 

48. Khosravi MR. Reliability of scholarly journal acceptance rates. Library Hi Tech News. 2018. https://doi.org/10.1108/LHTN-07-2018-0044.

49. Charles W, Fox AYK, Albert TH. Vines, Recruitment of reviewers is becoming harder at some journals: a test of the influence of reviewer fatigue at six journals in ecology and evolution. Res Integrity Peer Rev. 2017;2:3.

    Google Scholar 

50. Gropp RE, Glisson S, Gallo S, Thompson L. Peer review: a system under stress. BioScience. 2017;67:407–10.

    Google Scholar 

51. Kovanis M, Trinquart L, Ravaud P, Porcher R. Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication. Scientometrics. 2017;113:651–71.

    Google Scholar 

52. Heesen R, Romeijn J-W. Epistemic diversity and editor decisions: a statistical Matthew effect. Philosophers’ Imprint. 2019. http://philsci-archive.pitt.edu/16262/.

53. Hofmeister R, Krapf M. How do editors select papers, and how good are they at doing it? B.E. J Econ Analysis Policy. 2011;11. https://doi.org/10.2202/1935-1682.3022.

54. Morgan AC, Economou DJ, Way SF, Clauset A. Prestige drives epistemic inequality in the diffusion of scientific ideas. EPJ Data Sci. 2018;7:1–16.

    Google Scholar 

55. Dal-Ré R, Caplan AL, Marusic A. Editors’ and authors’ individual conflicts of interest disclosure and journal transparency. A cross-sectional study of high-impact medical specialty journals. BMJ Open. 2019;9:e029796.

    Google Scholar 

56. Teixeira da Silva JA, Dobránszki J, Bhar RH, Mehlman CT. Editors should declare conflicts of interest. Bioethical Inquiry. 2019;16:279–98.

    Google Scholar 

57. Huisman J, Smits J. Duration and quality of the peer review process: the author’s perspective. Scientometrics. 2017;113:633–50.

    Google Scholar 

58. A. Marusic, 10 The role of the peer review process. Fraud and Misconduct in Biomedical Research, 128 (2019).

59. N. van Sambeek, D. Lakens, “Reviewers’ decision to sign reviews is related to their recommendation” (preprint, PsyArXiv, 2019), , doi:https://doi.org/10.31234/osf.io/4va6p.

60. Bornmann L, Mutz R, Daniel H-D. A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants. PLOS ONE. 2010;5:e14331.

    Google Scholar 

61. Campos-Arceiz A, Primack RB, Koh LP. Reviewer recommendations and editors’ decisions for a conservation journal: is it just a crapshoot? And do Chinese authors get a fair shot? Biol Conservation. 2015;186:22–7.

    Google Scholar 

62. Tennant JP, Penders B, Ross-Hellauer T, Marušić A, Squazzoni F, Mackay AW, Madan CR, Shaw DM, Alam S, Mehmani B, Graziotin D, Nicholas D. Boon, bias or bane? The potential influence of reviewer recommendations on editorial decision-making. Eur Sci Editing. 2019;45. https://doi.org/10.20316/ESE.2019.45.18013.

63. Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer review. J Assoc Inform Sci Technol. 2013;64:2–17.

    Google Scholar 

64. Tennant JP. The dark side of peer review. Editorial Office News. 2017;10:2.

    Google Scholar 

65. Sandström U, Hällsten M. Persistent nepotism in peer-review. Scientometrics. 2008;74:175–89.

    Google Scholar 

66. Teplitskiy M, Acuna D, Elamrani-Raoult A, Körding K, Evans J. The sociology of scientific validity: how professional networks shape judgement in peer review. Res Policy. 2018;47:1825–41.

    Google Scholar 

67. Glonti K, Cauchi D, Cobo E, Boutron I, Moher D, Hren D. A scoping review protocol on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals. BMJ Open. 2017;7:e017468.

    Google Scholar 

68. Glonti K, Cauchi D, Cobo E, Boutron I, Moher D, Hren D. A scoping review on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals. BMC Med. 2019;17:118.

    Google Scholar 

69. M. Dahrendorf, T. Hoffmann, M. Mittenbühler, S.-M. Wiechert, A. Sarafoglou, D. Matzke, E.-J. Wagenmakers, “Because it is the right thing to do”: taking stock of the Peer Reviewers’ Openness Initiative” (preprint, PsyArXiv, 2019), , doi:https://doi.org/10.31234/osf.io/h39jt.

70. Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single- versus double-blind peer review. Proc Natl Acad Sci. 2017;114:12708–13.

    Google Scholar 

71. H. Bastian, The Fractured Logic of Blinded Peer Review in Journals (2017; http://blogs.plos.org/absolutely-maybe/2017/10/31/the-fractured-logic-of-blinded-peer-review-in-journals/).

72. Lundine J, Bourgeault IL, Glonti K, Hutchinson E, Balabanova D. “I don’t see gender”: conceptualizing a gendered system of academic publishing. Soc Sci Med. 2019;235:112388.

    Google Scholar 

73. Lynam DR, Hyatt CS, Hopwood CJ, Wright AGC, Miller JD. Should psychologists sign their reviews? Some thoughts and some data. J Abnormal Psychol. 2019;128:541–6.

    Google Scholar 

74. Baggs JG, Broome ME, Dougherty MC, Freda MC, Kearney MH. Blinding in peer review: the preferences of reviewers for nursing journals. J Advanced Nurs. 2008;64:131–8.

    Google Scholar 

75. J. Tóth, Blind myself: simple steps for rditors and software providers to take against affiliation bias. Sci Eng Ethics (2019), doi:10/gf6zzj.

76. Tennant JP. The state of the art in peer review. FEMS Microbiol Lett. 2018;365. https://doi.org/10.1093/femsle/fny204.

77. van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers’recommendations: a randomised trial. BMJ. 1999;318:23–7.

    Google Scholar 

78. Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D. Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. JAMA. 1998;280:240–2.

    Google Scholar 

79. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. A randomized trial. JAMA. 1990;263:1371–6.

    Google Scholar 

80. Okike K, Hug KT, Kocher MS, Leopold SS. Single-blind vs double-blind peer review in the setting of author prestige. JAMA. 2016;316:1315–6.

    Google Scholar 

81. Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. 1998;280:237–40.

    Google Scholar 

82. Bianchi F, Grimaldo F, Squazzoni F. The F3-index. Valuing reviewers for scholarly journals. J Informetrics. 2019;13:78–86.

    Google Scholar 

83. Cowley SJ. How peer-review constrains cognition: on the frontline in the knowledge sector. Front. Psychol. 2015;6. https://doi.org/10.3389/fpsyg.2015.01706.

84. J. P. Alperin, C. M. Nieves, L. Schimanski, G. E. Fischman, M. T. Niles, E. C. McKiernan, How significant are the public dimensions of faculty work in review, promotion, and tenure documents? (2018) (available at https://hcommons.org/deposits/item/hc:21015/).

85. Priem J, Hemminger BM. Decoupling the scholarly journal. Front Comput Neurosci. 2012;6. https://doi.org/10.3389/fncom.2012.00019.

86. Ghosh SS, Klein A, Avants B, Millman KJ. Learning from open source software projects to improve scientific review. Front Comput Neurosci. 2012;6:18.

    Google Scholar 

87. Horbach SPJM, Halffman W. The ability of different peer review procedures to flag problematic publications. Scientometrics. 2019;118:339–73.

    Google Scholar 

88. Superchi C, González JA, Solà I, Cobo E, Hren D, Boutron I. Tools used to assess the quality of peer review reports: a methodological systematic review. BMC Med Res Methodology. 2019;19:48.

    Google Scholar 

89. E. Adie, Commenting on scientific articles (PLoS edition) (2009), (available at http://blogs.nature.com/nascent/2009/02/commenting_on_scientific_artic.html).

90. Ginsparg P. ArXiv at 20. Nature. 2011;476:145–7.

    Google Scholar 

91. Morey RD, Chambers CD, Etchells PJ, Harris CR, Hoekstra R, Lakens D, Lewandowsky S, Morey CC, Newman DP, Schönbrodt FD, Vanpaemel W, Wagenmakers E-J, Zwaan RA. The Peer Reviewers’ Openness Initiative: incentivizing open research practices through peer review. Royal Soc Open Sci. 2016;3:150547.

    Google Scholar 

92. Fanelli D. How many scientists fabricate and falsify research? A systematic teview and meta-analysis of survey data. PLOS ONE. 2009;4:e5738.

    Google Scholar 

93. E. C. McKiernan, L. A. Schimanski, C. M. Nieves, L. Matthias, M. T. Niles, J. P. Alperin, “Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations” (e27638v2, PeerJ Inc., 2019), , doi:https://doi.org/10.7287/peerj.preprints.27638v2.

94. Schimanski LA, Alperin JP. The evaluation of scholarship in academic promotion and tenure processes: Past, present, and future. F1000Res. 2018;7:1605.

    Google Scholar 

95. Keserlioglu K, Kilicoglu H, ter Riet G. Impact of peer review on discussion of study limitations and strength of claims in randomized trial reports: a before and after study. Res Integrity Peer Rev. 2019;4:19.

    Google Scholar 

96. Danchev V, Rzhetsky A, Evans JA. Centralized scientific communities are less likely to generate replicable results. eLife. 2019;8:e43094.

    Google Scholar 

97. Kumar M. A review of the review process: manuscript peer-review in biomedical research. Biol Med. 2009;1:16.

    Google Scholar 

98. Campanario JM. Rejecting and resisting Nobel class discoveries: accounts by Nobel Laureates. Scientometrics. 2009;81:549–65.

    Google Scholar 

99. Neylon C, Pattinson D, Bilder G, Lin J. On the origin of nonequivalent states: How we can talk about preprints. F1000Res. 2017;6:608.

    Google Scholar 

100. E. Adie, Who comments on scientific papers – and why? (2008), (available at http://blogs.nature.com/nascent/2008/07/who_leaves_comments_on_scienti_1.html).

101. Ginsparg P. Preprint Déjà Vu. EMBO J. 2016:e201695531.

102. A. Gentil-Beccot, S. Mele, T. Brooks, Citing and reading behaviours in high-energy physics. How a community stopped worrying about journals and learned to love repositories. arXiv:0906.5418 [cs] (2009) (available at http://arxiv.org/abs/0906.5418).

103. Carneiro CFD, Queiroz VGS, Moulin TC, Carvalho CAM, Haas CB, Rayêe D, Henshall DE, De-Souza EA, Espinelli F, Boos FZ, Guercio GD, Costa IR, Hajdu KL, Modrák M, Tan PB, Burgess SJ, Guerra SFS, Bortoluzzi VT, Amaral OB. Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature. bioRxiv. 2019:581892.

104. Tennant JP, Bauin S, James S, Kant J. The evolving preprint landscape: Introductory report for the Knowledge Exchange working group on preprints. BITSS. 2018. https://doi.org/10.17605/OSF.IO/796TU.

105. Marra M. Astrophysicists and physicists as creators of ArXiv-based commenting resources for their research communities. An initial survey. Inform Services Use. 2017;37:371–87.

     Google Scholar 

106. S. Hindle, Saderi, PREreview — a new resource for the collaborative review of preprints (2017; https://elifesciences.org/labs/57d6b284/prereview-a-new-resource-for-the-collaborative-review-of-preprints).

107. T. Ross-Hellauer, B. Schmidt, B. Kramer, “Are funder Open Access platforms a good idea?” (PeerJ Inc., 2018), , doi:https://doi.org/10.7287/peerj.preprints.26954v1.

108. Moore SA. A genealogy of open access: negotiations between openness and access to research. Revue française des sciences de l’information et de la communication. 2017. https://doi.org/10.4000/rfsic.3220.

109. R. I. Network, Activities, costs and funding flows in the scholarly communications system in the UK: Report commissioned by the Research Information Network (RIN) (2008).

110. Stemmle L, Collier K. RUBRIQ: tools, services, and software to improve peer review. Learned Publishing. 2013;26:265–8.

     Google Scholar 

111. V. Demicheli, C. Di Pietrantonj, Peer review for improving the quality of grant applications. Cochrane Database Syst Rev, MR000003 (2007).

112. T. Jefferson, M. Rudin, S. Brodney Folse, F. Davidoff, Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev, MR000016 (2007).

113. Rennie D. Let’s make peer review scientific. Nat News. 2016;535:31.

     Google Scholar 

114. Squazzoni F, Ahrweiler P, Barros T, et al. Unlock ways to share data on peer review. Nature. 2020;578:512–4. https://doi.org/10.1038/d41586-020-00500-y.

     Article  Google Scholar 

115. Ioannidis JPA, Berkwits M, Flanagin A, Godlee F, Bloom T. Ninth international congress on peer review and scientific publication: call for research. BMJ. 2019;366. https://doi.org/10.1136/bmj.l5475.

Page 2

1. The difficulty levels primarily refer to the relative ease of obtaining empirical data for study, should such data even exist. The priority levels relate to their perceived impact on the future of peer review. Both are subjective estimates of the authors