F1000Research Open Peer Review RESEARCH NOTE An Open Science Peer Review Oath [v2; ref status: indexed, http://f1000r.es/4wf] Jelena Aleksic , Adrian Alexa , Teresa K Attwood , Neil Chue Hong , Martin Dahlö , Robert Davey , Holger Dinkel , Konrad U Förstner , Ivo Grigorov , Jean-Karim Hériché , Leo Lahti , Dan MacLean , Michael L Markie , Jenny Molloy , Maria Victoria Schneider , Camille Scott , Richard Smith-Unna , Bruno Miguel Vieira , as part of the AllBio: Open Science & Reproducibility Best Practice Workshop Wellcome Trust – Medical Research Council Cambridge Stem Cell Institute, University of Cambridge, Cambridge, CB2 1QR, UK DNAdigest, Cambridge, UK University of Manchester, Manchester, UK Science for Life Laboratory, Uppsala University, Uppsala, Sweden The Genome Analysis Centre, Norwich Research Park, Norwich, NR4 7UH, UK European Molecular Biology Laboratory, Heidelberg, Germany Core Unit Systems Medicine, University of Würzburg, Würzburg, Germany DTU Aqua, Technical University of Denmark, Charlottenlund 2920, Denmark Software Sustainability Institute, Edinburgh, UK Open Knowledge Finland - Open Science Work Group, Helsinki, Finland The Sainsbury Laboratory, Norwich Research Park, Norwich, NR4 7UH, UK F1000Research, London, UK Department of Zoology, University of Oxford, Oxford, UK Michigan State University, East Lansing, MI, USA Department of Plant Sciences, University of Cambridge, Cambridge, UK School of Biological and Chemical Sciences, Queen Mary University of London, London, UK Abstract One of the foundations of the scientific method is to be able to reproduce experiments and corroborate the results of research that has been done before. However, with the increasing complexities of new technologies and techniques, coupled with the specialisation of experiments, reproducing research findings has become a growing challenge. Clearly, scientific methods must be conveyed succinctly, and with clarity and rigour, in order for research to be reproducible. Here, we propose steps to help increase the transparency of the scientific method and the reproducibility of research results: specifically, we introduce a peer-review oath and accompanying manifesto. These have been designed to offer guidelines to enable reviewers (with the minimum friction or bias) to follow and apply open science principles, and support the ideas of transparency, reproducibility and ultimately greater societal impact. Introducing the oath and manifesto at the stage of peer review will help to check that the research being published includes everything that other researchers would 1 2 3 9 4 5 6 7 8 6 10 11 12 13 5 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Referee Status: Invited Referees 12 Nov 2014, :271 (doi: )First published: 3 10.12688/f1000research.5686.1 09 Jan 2015, :271 (doi: )Latest published: 3 10.12688/f1000research.5686.2 v2 Page 1 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015 F1000Research , University of New York atVitaly Citovsky Stony Brook USA , Cardiff University UKChris Chambers , IPBS CNRS Université PaulEtienne Joly Sabatier France , Stony Brook UniversitySuzanne Scarlata USA , University of PittsburghLawrence P Kane USA Discuss this article (3)Comments 5 4 3 2 1 the oath and manifesto at the stage of peer review will help to check that the research being published includes everything that other researchers would need to successfully repeat the work. Peer review is the lynchpin of the publishing system: encouraging the community to consciously (and conscientiously) uphold these principles should help to improve published papers, increase confidence in the reproducibility of the work and, ultimately, provide strategic benefits to authors and their institutions. Dan MacLean ( )Corresponding author: Dan.MacLean@sainsbury-laboratory.ac.uk Aleksic J, Alexa A, Attwood TK How to cite this article: et al. An Open Science Peer Review Oath [v2; ref status: indexed, 2015, :271 (doi: )]http://f1000r.es/4wf F1000Research 3 10.12688/f1000research.5686.2 © 2015 Aleksic J . This is an open access article distributed under the terms of the , whichCopyright: et al Creative Commons Attribution Licence permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Data associated with the article are available under the terms of the (CC0 1.0 Public domain dedication).Creative Commons Zero "No rights reserved" data waiver ALLBIO - Broadening the Bioinformatics Infrastructure to unicellular, animal, and plant science, Project reference: 289452,Grant information: Funded under: FP7-KBBE. We would also like to thank The Genome Analysis Centre (TGAC, Norwich, UK) and the Biotechnology and Biological Sciences Research Council (BBSRC, UK). IG was funded by FP7 FOSTER (Grant 612 425). Competing interests:MM is currently employed by . His role at the journal does not include any involvement in theF1000Research pre-publication editorial checks, or with the refereeing process. 12 Nov 2014, :271 (doi: ) First published: 3 10.12688/f1000research.5686.1 09 Dec 2014, :271 (doi: )First indexed: 3 10.12688/f1000research.5686.1 version 2 published 09 Jan 2015 version 1 published 12 Nov 2014 1 2 3 4 5 report report report report report report Page 2 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015 Open science and the future of peer-review Science builds on itself. New knowledge is gained in the context of the enlightenment of earlier discoveries and, for the founda- tions to remain solid each discovery must be accurate and reliable. But science in the real world is messy, and advances haltingly and piecewise. Often, prior information is incomplete, so conclusions drawn need revising in the light of new, later evidence. This means that self-reference and self-checking are cornerstones of the scien- tific method; and, having reported our experiments, it is vital that readers are able to repeat them and understand how we reached our conclusions. Various studies have shown that, for a study to be successfully reproduced, information presented in the literature is often inadequate and the underlying data not readily available1 - a significant drawback. Several commentators have concluded that weaknesses in the way that research investigations are currently conducted, and how their results are disseminated via article publi- cation have become detrimental to the scientific process2–7. As the technological sophistication of science increases, and the equip- ment used becomes more specialised, the data generated are harder to represent in traditional media, and reporting how experiments were performed so that independent researchers can repeat them becomes progressively harder. Open science is a movement that seeks to ensure that the results and the data of scientific research are, and continue to be, avail- able to all. One way in which reproducibility issues can be tack- led is through the use of open-science and open-data practices8,9. As attendees of the AllBio: Open Science & Reproducibility Best Practice Workshop, we discussed how the problem of keeping sci- ence transparent and reproducible in an increasingly technology- driven, and specialised, domain could be addressed. One route, at the heart of scientific endeavour, is through the peer- review process. Peer review is an important gatekeeper and key component of scientific discourse. Before research findings can be formally accepted, they must be evaluated and commented upon by other domain experts, who provide advice about the quality or validity of the work to journal editors and/or readers (whether via pre-publication peer review, open peer review or post-publication peer-review systems). Importantly, peer review happens at a personal, rather than institutional, level and is carried out by individuals; it is therefore an ideal mechanism for engaging most researchers, given that all scientists peer review or are peer reviewed. Peer review oaths The peer-review process is not infallible10–12; its weaknesses are many and varied, including the time available to perform thorough reviews, reviewers’ expertise, journals’ perception of relevance/ interest/impact, and so on. Arguably, one of the most significant problems – certainly the one that generates most friction – is that reviewers can safely dispense self-serving and biased critiques, fully protected by the mask of anonymity. In some instances, anonymity may be the only pragmatic option, but in the interest of objectivity and constructive discourse it shouldn’t be the cor- nerstone of the peer-review process. Scientists have become suf- ficiently frustrated by these issues to devise ad hoc solutions to help safeguard the quality of reviews, and allow reviewers to affirm that they will review in an ethical and professional way, and encourage clearer review processes. This has led to the articulation of various reviewer’s oaths (e.g.,13–15); it is these that inspired our discussions and prompted this article. An outcome of our debate was the formulation of an addition to these oaths that helps codify the role of reviewers, encouraging and promoting best practices to ensure that the science they review is as open and reproducible as possible. The amendment includes guide- lines not only on how to review professionally, but also on how to support transparent, reproducible and responsible research, while optimising its societal impact and maximizing its visibility. The open-science peer-review oath (which can be found on figshare for reuse16) we therefore propose is as follows (Box 1). Box 1. Open peer review oath. Principle 1: I will sign my name to my review Principle 2: I will review with integrity Principle 3: I will treat the review as a discourse with you; in particular, I will provide constructive criticism Principle 4: I will be an ambassador for the practice of open science Towards ‘open science friendly’ reviews The declaration in your review Peer-review oaths tend to be short declarations that reviewers make at the start of their written comments, typically dictating the terms by which they will conduct their reviews. The novelty in our amendment is the addition of a new declaration, which we recom- mend including in review preambles: I will be an ambassador for open science For it to be successful requires reviewers to follow the spirit of the statement throughout their reviews, which may involve a little addi- tional work. The statement itself is derived from the rationale out- lined below. It is for reviewers to decide how to implement this on a case-by-case basis – it is case specific, but nevertheless provides a framework for nurturing useful and hopefully fruitful discussions. Guidelines for open science reviewers Here is how we envisage the review process from an open-science reviewer’s point of view: • I will work with you to help improve your research, as I believe that peer review should be an open, supportive and collaborative process. I will therefore sign my review and state my identity. Amendments from Version 1 We have taken into consideration all of the reviews and comments, and have made the manuscript shorter and clearer. We have further discussed what we hope to achieve from researchers using the oath, and we have balanced the article to explain how we believe the oath can encourage the practice of open science. The oath itself now comprises four concise principles, and we elucidate the open-science principle with accompanying guidelines, which should make it easier to apply in practice. See referee reports REVISED Page 3 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015 • I will, before I review, ensure that I have everything (raw data, detailed methods/protocols, etc.) necessary to make a complete, independent assessment. • I will encourage the application, and, where necessary, provide guidance for any open-science best practices rel- evant to my field that would help support transparency, reproducibility, re-use and integrity of your research. • I will check that any data and software code are consist- ent with the text, that any digital object identifiers and accession numbers are correct and correctly cited, and that any models presented are archived, referenced and accessible. • I will check that the data and any software code and sup- port documentation that underpin the published concept are made available in a manner that provides long-term unrestricted access. • I will advise, where necessary, on how to achieve bet- ter transparency and availability (in terms of materi- als and methodology, data and code access, versioning, algorithms, software parameters and standards), with the understanding that adherence to best open-science prac- tices may require further effort from you. • I will support others in writing open reviews, where it is appropriate for me to do so. • I will decline to review if I am not an appropriate reviewer (whether because of my expertise or because of my rela- tionship with the author(s)). In doing so, I will provide journal editors with an honest appraisal of these issues, and will openly explain how I reached my decision so that alternative reviewers may be found. Conclusion We believe that combining the open-science principle with some of the other key peer-review principles will help the scientific com- munity to repeat published experiments, and to reach the same, or similar, conclusions. Author contributions Dan Maclean, Ivo Grigorov, Michael Markie, Teresa Attwood, Konrad Förstner, Jean-Karim Heriche and Neil Chue Hong con- ceived and designed the oath and prepared the first draft of the man- uscript. All the other authors in the working group were involved in the revision of the draft manuscript and have agreed to the final content. Competing interests MM is currently employed by F1000Research. His role at the jour- nal does not include any involvement in the pre-publication edito- rial checks, or with the refereeing process. Grant information ALLBIO - Broadening the Bioinformatics Infrastructure to unicellu- lar, animal, and plant science, Project reference: 289452, Funded under: External link: FP7-KBBE. We would also like to thank The Genome Analysis Centre (TGAC, Norwich, UK) and the Biotech- nology and Biological Sciences Research Council (BBSRC, UK). Acknowledgements We would like to thank The Genome Analysis Centre (TGAC, Norwich, UK) for organising and hosting the workshop. We would also like to thank Peter Murray Rust for looking over the preprint of the manuscript (https://zenodo.org/record/12273) and contributing an additional principle to the oath. References 1. Collins FS, Tabak LA: Policy: NIH plans to enhance reproducibility. Nature. 2014; 505(7485): 612–3. PubMed Abstract | Publisher Full Text | Free Full Text 2. Ioannidis JP: Why most published research findings are false. PLoS Med. 2005; 2(8): e124. PubMed Abstract | Publisher Full Text | Free Full Text 3. Ioannidis JP, Allison DB, Ball CA, et al.: Repeatability of published microarray gene expression analyses. Nat Genet. 2009; 41(2): 149–55. PubMed Abstract | Publisher Full Text 4. Begley GC, Ioannidis JP: Reproducibility in Science: Improving the Standard for Basic and Preclinical Research. Circ Res. 2015; 116(1): 116–126. PubMed Abstract | Publisher Full Text 5. Prinz F, Schlange T, Asadullah K: Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov. 2011; 10(9): 712. PubMed Abstract | Publisher Full Text 6. Hines WC, Su Y, Kuhn I, et al.: Sorting out the FACS: a devil in the details. Cell Rep. 2014; 6(5): 779–81. PubMed Abstract | Publisher Full Text 7. Vasilevsky NA, Brush MH, Paddock H, et al.: On the reproducibility of science: unique identification of research resources in the biomedical literature. PeerJ. 2013; 1: e148. PubMed Abstract | Publisher Full Text 8. Molloy JC: The Open Knowledge Foundation: open data means better science. PLoS Biol. 2011; 9(12): e1001195. PubMed Abstract | Publisher Full Text | Free Full Text 9. Pereira S, Gibbs RA, McGuire AL: Open access data sharing in genomic research. Genes (Basel). 2014; 5(3): 739–747. PubMed Abstract | Publisher Full Text | Free Full Text 10. Patel J: Why training and specialization is needed for peer review: a case study of peer review for randomized controlled trials. BMC Med. 2014; 12(1): 128. PubMed Abstract 11. Glen AS: A New “Golden Rule” for Peer Review? Bull Ecol Soc Am. 2014; 95(4): 431–434. Publisher Full Text 12. Siler K, Lee K, Bero L: Measuring the effectiveness of scientific gatekeeping. Proc Natl Acad Sci U S A. 2014; pii: 201418218. PubMed Abstract | Publisher Full Text 13. Watson M: The reviewer’s oath. 2013. Reference Source 14. Alexander S: The Peer Reviewer’s Oath. 2014. Reference Source 15. Verger A: My Reviewer Oath. 2014. Reference Source 16. MacLean D, Aleksic J, Attwood TK, et al.: Open Science Peer Review Oath. figshare. 2014. Publisher Full Text Page 4 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015 F1000Research Open Peer Review Current Referee Status: Version 2  12 January 2015Referee Report doi:10.5256/f1000research.6351.r7276 Vitaly Citovsky Department of Biochemistry and Cell Biology, University of New York at Stony Brook, Stony Brook, NY, USA The revised paper is much improved in terms of its (now appropriate and not tedious) length and clarity of statements. I still see a problem with the proposed absolute requirement (listed as the first principle) of disclosing the reviewer's identity. While noble in principle, I think it is unrealistic at this time; if so, compromising the first principle would appear to make the subsequent ones partly irrelevant. On the other hand, the authors clearly explained their decision to avoid addressing one specific issue of the reviewing process - frequent requests for tedious and time consuming experiments that do not contribute substantially to the main point of the paper, often precluding publication. As an experimental scientist, however, I do see this point as critical and think that its resolution would contribute more to the fairness of review than disclosing the reviewer's identity. Finally, I am still surprised that the related work by Bienz and Weston is not discussed and/or cited. I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. No competing interests were disclosed.Competing Interests: Version 1  10 December 2014Referee Report doi:10.5256/f1000research.6078.r6800 Lawrence P Kane Department of Immunology, University of Pittsburgh, Pittsburgh, PA, USA The principles outlined here are important, and this piece is certainly timely. I have two suggestions to improve the manuscript. First, as currently constituted, the manuscript is a bit repetitive, with a large text box recapitulating what is also laid out in much the same format in the body of the manuscript. Perhaps each of the major points could be expounded upon in the text. Page 5 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015 F1000Research each of the major points could be expounded upon in the text. Second, while it is hard to argue with the specific tenets enumerated here, there are quite a lot of them. I guess my feeling is that for ideas like this to be more widely adopted that it would be helpful if the core principles could be boiled down to a more manageable size (which would be ideally presented in a text box). I applaud the authors for doing their part to increase openness in scientific publishing, something I agree is very much needed. I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. I am a Section Editor for the Journal of ImmunologyCompeting Interests: Author Response 10 Jan 2015 , The Sainsbury Laboratory, UKDan MacLean Thanks for your helpful and candid report Larry. We agree the manuscript was a little repetitive and so we have made it much more succinct, by concentrating on the open science principle we want to champion. We have also condensed the oath into 4 principles with an accompanying rationale and made it available on FigShare, which should make it much easier to follow and reuse. No competing interests were disclosed.Competing Interests:  09 December 2014Referee Report doi:10.5256/f1000research.6078.r6987 Suzanne Scarlata Department of Physiology and Biophysics, Stony Brook University, Stony Brook, NY, USA I think this article makes many good points, and I also agree with the other critiques. However, I do see danger in full transparency. The problem is that significance of study can be subjective and used in a biased way to sway readers towards a view of greater importance, in order to achieve a good standing with the authors. At this point in time, it is important for editors of other non-transparent journals to remind reviewers of the key aspects of this oath – to critically read the paper and make constructive comments as best as they can. Editors need to use their power to delete inappropriate reviews or reviewers. I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. No competing interests were disclosed.Competing Interests: Author Response 10 Jan 2015 , The Sainsbury Laboratory, UKDan MacLean Page 6 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015 F1000Research 1. 2. 3. Thanks Suzanne for your helpful and candid report. We have now satisfied the other reviews to make the oath clearer and simpler to use. We take your point that perceived impact can indeed sway readers and we certainly agree that as well as the scientific community practicing open science and reviewing openly, journals and journal editors need to play a role in ensuring that reviewers are doing a job that ensures integrity - we believe they can use the principles of the oath to do this. No competing interests were disclosed.Competing Interests:  27 November 2014Referee Report doi:10.5256/f1000research.6078.r6802 Etienne Joly Equipe de Neuro-Immunogénétique Moléculaire (ENIGM), Bâtiment CNRS, IPBS CNRS Université Paul Sabatier, Toulouse, France This manuscript was written by participants of a workshop entitled "AllBio: Open Science & Reproducibility Best Practice Workshop" which took place at the TGAC in Norwich UK in September 2014. The alleged purpose of the oath and manifesto proposed in this manuscript is to make "transparency, reproducibility and citizen-scientist engagement the default parameters for performing sound research." Those goals are clearly highly laudable, and I completely agree that the process of scientific refereeing would greatly improve by going 'open'. I do, however, feel very ill at ease about acting as a referee for this particular paper, for the following three main reasons: This is not a scientific manuscript, and it neither contains data, nor reviews a scientific topic. I am thus left wondering why such a manuscript should need to be peer reviewed. After reading it, I certainly cannot conclude that it is scientifically sound. The best I can conclude is that it is not scientifically unsound. Although this manuscript is very short, I must say that I found it rather difficult to read because of its structure which contains many redundancies and of the fuzziness of its purpose. One aspect that particularly bothers me is the unquestioned assumption that open refereeing will improve reproducibility. The first thing the authors should clarify is what they mean by reproducibility since there are at least two types that I can think of: The first one concerns data reproducibility (or robustness). In other words will similar data, leading to the same conclusions, be obtained if the experiment is repeated (on a different day and/or with different samples and/or in a different place and/or by different people etc…). The second type of reproducibility relates to the capacity of other scientists to reproduce an experiment described in a published manuscript. I suspect that when they refer to 'science reproducibility', the authors refer to the latter type, although the first type is the most important one in my eyes. The authors duly acknowledge that they have been inspired by reviewer's oaths previously proposed by others (refs 13-15), and if I am being completely honest, I do not find that the set of 17 rules they propose represent a significant improvement on those previously proposed oaths. Because I do not want to plagiarise Jonathan Eisen, I will simply suggest that the authors should Page 7 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015 F1000Research 3. 1. 2. Because I do not want to plagiarise Jonathan Eisen, I will simply suggest that the authors should very seriously consider following the many suggestions he has made on his blog to improve this paper ( ), and especially the idea of reversing the Oath and thehttp://icis.ucdavis.edu/?p=505 Manifesto. I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. No competing interests were disclosed.Competing Interests: Author Response 10 Jan 2015 , The Sainsbury Laboratory, UKDan MacLean Thanks Etienne for your very helpful and candid report. We agree that the article needed to be made clearer in order to convey its intentions to the reader and we have taken measures to do this by clearly elucidating the open science principle. We have also made it clear about what we mean by reproducibility, and how the robustness of research should mean that similar data should be able to produce the same conclusions. We have also simplified the oath on the advice given to us from you and the other reviewers to make it easier to read and reuse. No competing interests were disclosed.Competing Interests:  26 November 2014Referee Report doi:10.5256/f1000research.6078.r6797 Chris Chambers School of Psychology, Cardiff University, Cardiff, UK Overall, I believe this is a laudable proposal for a code of practice in academic peer review. In one sense it would be nice if such a code was unnecessary; after all, most of the practices outlined here should form part of any graduate training in science. In practice, of course, we know that the reality of peer review often falls short of achieving its aims. Here the authors outline 5 key principles and 17 key practices, which - if adhered to - would likely result in a more transparent and effective peer review mechanism. While I am supportive of this initiative, I do have some concerns about the way it is presented and I also wonder how adhering to it could be incentivised. I will outline below some suggestions for possible improvement: The authors begin by outlining the 17 key practices and only at the end do they group them together within governing principles. I think the overall structure of the paper would be clearer if this order were reversed - beginning with the five principles, explaining the key practices in each case that serve them, before returning to the code of practice or "oath" as the authors call it. Some of the individual practices in the oath are somewhat ambiguous to me, and others seem to overlap. For instance, what exactly does it mean for a reviewer to "state their limits"? This is elaborated later under principle 2 but I would recommend clearer descriptions at all times. Other practices seem very similar, or least belong to common subsets of behaviour; e.g. practice 7 and Page 8 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015 F1000Research 2. 3. 4. 5. practices seem very similar, or least belong to common subsets of behaviour; e.g. practice 7 and practice 10; practice 2 and 6; practices 13, 14, 15 and 16 are all very similar and not clearly distinguished. With some careful attention I suspect the number of practices here could be halved, which would improve readability and likely uptake. The language expanding on some of the core principles is perhaps little purple in places, e.g. "I will I'm notuse the majority of ‘doves’ to balance the ‘hawks’ in my review by sharing the content." entirely sure what this means, but if it refers to the intention to balance sharp critical feedback with constructive suggestions or positive feedback, this could perhaps we said in a more straightforward way. The authors should bear in mind that not all readers will have English as a first language, and so more direct and less metaphorical phrasing could widen the appeal. Two concerns with signed reviews need to be addressed. The first is the potential negative consequences felt by junior (non-tenured) scientists or minorities, who could, at least in theory, face severe repercussions for criticizing the work of senior/powerful colleagues who sit on editorial boards or grant panels. In my mind this conflict has never been properly addressed; while I believe it is completely reasonable for tenured scientists to be open and accountable in their reviews (I always sign mine), it is questionable whether this should be required uniformly across science. The second concern with open reviewing is the potential legal backlash of scientists being sued by litigious authors who feel aggrieved by a reviewer's published comments. We are seeing this already on PubPeer and elsewhere. This raises the question of who assumes legal responsibility for the content of a signed review. In the case of , for instance, does it or will itF1000Research provide legal indemnity to reviewers who choose to unmask themselves? Finally, I would prompt the authors of this paper to consider, and ideally speculate on, ways their oath (or code of practice as I would call it) might be implemented and incentivised in science. Could it, for instance, be worked into the next REF in some way? How could this be achieved and what challenges would need to be overcome? How does this initiative relate to other emerging group-led initiatives, such as the Agenda for Open Research (https://agendaforopenresearch.org/)? There authors have no shortage of good intentions but as we know, there lies a world of groupthink and social inertia between good intentions and good practices. Regardless of the above concerns, I applaud this much-needed call for greater transparency in the peer review process. I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. No competing interests were disclosed.Competing Interests: Author Response 09 Jan 2015 , F1000Research, UKMichael Markie Thanks Chris for your very helpful and candid report. Like all of the reviewers have suggested we have simplified the oath and made it much clearer and simpler to use. Yes, we agree that some of the language used was difficult for non-native English speakers and hence we have made it clearer to read in places that may of caused confusion. We also agree with your point about new/junior reviewers who may feel exposed by openly reviewing someone more senior than them and have acknowledged that in the guidelines. Here at we have implemented a system ofF1000Research, Page 9 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015 F1000Research acknowledged that in the guidelines. Here at we have implemented a system ofF1000Research, co-authoring referee reports to help support junior researchers who feel that on their own they may be subject to repercussions from the authors. With regards to the legality of an open review on , reviewers submit their report according to our terms and conditions which clearlyF1000Research state the report will be published under a CC BY license, and that the report must not be defamatory. However as a publisher we are also conscious about each review that is published on the site and we have system in place to ensure that nothing libel would be intentionally published. More generally, openness and transparency in signing a report means that they are on the whole much more civil and are of a constructive nature that encourages dialogue. We believe the peer review process should be a collaborative process and hence that is the environment we try to provide. Here is a letter of response from Rebecca Lawrence, Managing Director of , recently published in response to the recent libel story from the anonymousF1000Research review on PubPeer: .http://www.timeshighereducation.co.uk/comment/letters/an-anonymity-problem/2017051.article Finally, just after we published the review we became aware of the Agenda for Open Research who have similar goals that align with us and they were kind enough to link to the oath in their guideline for reviewers. We fully support their initiative, and we will seek opportunities to collaborate where possible. You are right, it would be great to push to get open reviewing to be recognised as an official part of the REF (and other funding bodies) and acknowledged as a measurable scientific output. Here at we are taking steps to make this happen;F1000Research currently we mint each report with a DOI and we are heavily involved in the Peer Review Service in collaboration with and . project ORCID CASRAI I am the associate publisher of F1000Research.Competing Interests:  25 November 2014Referee Report doi:10.5256/f1000research.6078.r6804 Vitaly Citovsky Department of Biochemistry and Cell Biology, University of New York at Stony Brook, Stony Brook, NY, USA This article addresses a very important issue of peer review. Although, many of us tend to regard it in a way of "if it ain't broke, don't fix it", we also are often frustrated with the process and wish it were more fair, transparent and to the point (rather than requesting numerous experiments that are only tangential to the paper's main message). Thus, constructive approach to potential improvement of the peer review system is always welcome and should be encouraged. With this in mind, I support, in principle, publication of this manuscript. On the other hand, the way that the authors present their ideas is not perfect. Here is why: First they make a huge emphasis in their introductory remarks on experimental reproducibility and societal impact. However, their "oath" does not address these issues specifically and constructively; instead, it offers declarative statements which are mostly trivial and already represent part and parcel of today's peer review process (e.g., "I will comment on how well you have achieved transparency, in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards, such that your experiments can be repeated independently"). Second, the authors do not address clearly one very important aspect of potential improvement of the review process which is "To request further experiments only as a last resort, and only if they are essential Page 10 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015 F1000Research review process which is "To request further experiments only as a last resort, and only if they are essential to validate the conclusions of the paper. No experiments extending the study beyond its conclusions, or with unreasonable cost or time implications, should be proposed. An estimate of time required for the additional work should be provided." This is a quote from Mariann Bienz and Kathy Weston who also addressed the issue of peer reviewing ( ).http://elifesciences.org/elife-news/a-reviewers-charter Third, the "oath" is unreasonably long. Again, I suggest to look at the post by Bienz and Weston, which unlike the present paper, is much clearer, laconic and to the point. Fourth, the authors quite nonchalantly suggest to overturn the cornerstone of the present review system - its anonymity. What is the chance for that? The reason for such a step should be well reasoned and substantiated. I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. No competing interests were disclosed.Competing Interests: Author Response 10 Jan 2015 , The Sainsbury Laboratory, UKDan MacLean Thanks Vitaly for your very helpful and candid report. We have now explained what we hope to achieve from the oath, how we think it could address the issue of reproducibility and how the oath could have a positive societal impact in the scientific community. We believe at the researcher level there is a unique opportunity to help spread open science practices when reviewing articles and help improve what the community as a whole believes should be available in a paper in order for it to be reproduced. We agree with your point about not asking for additional experiments, however we have decided to concentrate fully on elucidating the requirements of the open science principle specifically and not the other principles. As with the other reviewers and comments we have taken all of your advice and shortened the oath to make it simpler for reuse. We also agree with your point about anonymity and we have discussed this in the manuscript on why we should move towards open peer review. No competing interests were disclosed.Competing Interests: Discuss this Article Version 2 Reader Comment 12 Jan 2015 , University of Chicago, USADaniel S. Katz I think this version is a good improvement - it removes most of the extra work by the reviewer that seemed to be implied in the previous version, and the extra work that is now left makes sense as a minimum. However, you may still want to consider that the needed functions of the review process do not all need to Page 11 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015 F1000Research However, you may still want to consider that the needed functions of the review process do not all need to be done by all reviewers, as long as all the functions are done. For example, perhaps 2 or 3 reviewers need to do the checks (of the data and software), not all need to do this. This might further reduce the reviewing burden. noneCompeting Interests: Author Response 12 Jan 2015 , The Sainsbury Laboratory, UKDan MacLean Thanks for your comment Daniel, we appreciate your thoughts. We have now simplified the principles of the oath and devised guidelines that concentrates solely on the open science principle. We do however agree that some of the points in our guidelines could be phrased to make them seem more “active” so we have taken in account your suggestions and we have also included a sentence in the manuscript that explains providing an open science review may well incur additional work for the reviewer. No competing interests were disclosed.Competing Interests: Author Response 12 Jan 2015 , The Sainsbury Laboratory, UKDan MacLean Thanks Jonathan for your very helpful and candid thoughts. We agree with you that the oath was indeed too long and not as impactful as it could be. For this reason we have taken your advice and provided a shorter more impactful oath of 4 principles with accompanying guidelines that specifically explain the inclusion of the open science principle. We have also swapped the roman numerals for numbers – in hindsight it is a lot simpler that way. We hope that this shorter version of the oath can now have the desired impact we want to achieve. No competing interests were disclosed.Competing Interests: Version 1 Reader Comment 22 Dec 2014 , University of Chicago, USADaniel S. Katz The suggested oath, broken down by principles and items, is quite interesting, with some of the principles having clear value and little overhead, while others that have potential value also add large amounts of possible overhead. I suggest that these be broken down into two types of open reviews. Specifically: :Passive review Follows all of Principles 1 through 3, and elements x - xii of Principle 4. Here, the idea of an open review basically is a review that is signed to make sure the process has appropriate integrity. :Active review Page 12 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015 F1000Research :Active review Includes all elements of Passive review, as well as items xiii and xiv of Principle 4: xiii) I will check that the data, software code and digital object identifiers are correct, and the models presented are archived, referenced, and accessible xiv) I will comment on how well you have achieved transparency, in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards, so that your experiments can be repeated independently The idea of an open review here seems to be that, in addition to ensuring integrity in the review process, there is also extra work being done beyond a standard review, and the person who does such work should be credited for doing so. I am uncertain about where elements xv and xvi, as written: xv) I will encourage deposition with long-term unrestricted access to the data that underpin the published concept, towards transparency and re-use; xvi) I will encourage central long-term unrestricted access to any software code and support documentation that underpin the published concept, both for reproducibility of results and software availability would fit. These are neither active nor passive, and as written, they don't match the review function, but are even more active, more a collaboration than a review. I suggest that they be rephrased as: xv) I will check that the data that underpin the published concept are made available in a manner that provides long-term unrestricted access, towards transparency and re-use; xvi) I will check that any software code and support documentation that underpin the published concept are made available in a manner that provides long-term unrestricted access, both for reproducibility of results and software availability so that they could be part of an Active review. Of course, this specific remedy is just a suggestion, but I the overall point I want to make is that the added work to be done by the reviewer beyond what is now standard needs to be explicitly considered in both the oath itself as well as the description of the oath. noneCompeting Interests: Reader Comment ( and ) 13 Nov 2014Member of the F1000 Faculty F1000Research Advisory Board Member , University of California Davis Medical Center, USAJonathan Eisen I have written a mini review of the paper on the UC Davis "Innovating Communication in Scholarship" blog: .http://icis.ucdavis.edu/?p=505 I am an informal/formal advisor to F1000 on some of their open science activitiesCompeting Interests: though I had no role in this paper. Page 13 of 13 F1000Research 2015, 3:271 Last updated: 13 JAN 2015