Let’s open up peer review

Photo by Mike Petrucci on Unsplash

I owe my career to a letter to the editor.

In 2003 the American epidemiologist Muin Khoury, together with several colleagues, published one of the first articles on how to predict diseases that are not caused by one DNA mutation but influenced by tens or hundreds of DNA variations. Think of common diseases such as type 2 diabetes, asthma, and cardiovascular diseases.

The formulas they proposed were valid, but I thought their assessment of how well these polygenic risk scores could predict was incorrect. I evaluated the risk scores using best practices and found less rosy results. Together with colleagues, I sent the journal a long letter that listed point by point what we thought was wrong with the article.

We didn’t hear from the journal for a long time, and I began to fear that the authors would burn us down mercilessly. I was new in science. I had received my PhD a few months before, had three publications, and had just started my first job as a postdoc. Muin Khoury was a pioneer in the new field of public health genomics and far ahead of everyone else.

Surprising and impressive
But the response of Khoury and his group was as surprising as it was impressive. They simply agreed with us. And they immediately outlined the steps for further research. As befits a pioneer.

Shortly after our letter and their response were published together, I was going to attend a conference in Atlanta, Georgia, Khoury’s hometown. I wrote him an e-mail, and he invited me to his institute, the Centers for Disease Control and Prevention. Khoury turned out to be just as friendly during our meeting as in his written response. He said with a smile how they had scratched their heads about how to respond, but that they could not deny that we had a point. “Apparently you know something that we want to know. Let’s collaborate.”

Khoury’s reaction made a deep impression on me at the time. This was how you deal with criticism. Even if you are the pope of your field, if someone is right, then someone is right. Acknowledge it openly and take advantage of it. How hard can it be?

Apparently very hard. Many researchers reject a system in which scientific articles are openly critiqued, with those critiques visible to everyone. Without this visibility, however, readers do not know how rigorously the article was evaluated, or if the right experts were even part of the review process.

Critical questions
And that goes to the heart of the scientific publication system. If the media mentions that a study has been published in The Lancet, Science, or another reputable scientific journal, then we assume that the research is okay. Because the quality has been ‘peer-reviewed’ by fellow scientists who have asked critical questions about the design, conduct, and interpretation of the research.

Unfortunately, ‘peer-reviewed’ does not always mean that an article has passed a serious quality check. The quality of peer review varies, and the joint expertise of the reviewers often does not cover all relevant aspects of a study. Some reviewers superficially reflect on the global content, others only check the statistics or readability of the article. But who reviewed the article with what expertise is unknown to other researchers and readers.

Peer review is usually anonymous. Authors do not see the names of their reviewers. Some journals go one step further, so the reviewers do not know whose work they critique — double-blind peer review.

Peer review is anonymous because reviewers can be more honest and critical without having to fear the consequences of negative reviews. Such consequences can include revenge, a negative assessment in the future when the roles are reversed. Especially for young researchers, this could have major consequences for their career.

Photo by chris panas on Unsplash

Lack of transparency
But I am not in favor of anonymous peer review. I don’t think the potential risks outweigh the consequences of a lack of transparency. Not anymore. There is much concern about the quality of research publications, about their validity and reproducibility. Trust in science depends on the quality of research and the accurate reporting thereof. And transparency is essential to guarantee the rigor of that quality assessment.

I prefer open peer review in which the peer reviews and the authors’ responses to them are published as an attachment to the article. This way, everyone can read how rigorous and complete the peer review was, how constructively the reviewers contributed to improving the article, and what the authors did with their suggestions, including their justifications.

Open peer review is like writing a letter to the editor, but in the pre-publication stage, when the authors still can correct any errors identified. Reviewers must stand behind their reviews, demonstrating publicly that they know what they are talking about and justifying their critiques. I find it hard to imagine how a respectful critical reflection can have negative consequences. It can even lead to a beautiful career.

I am hopeful that open peer review will be embraced as the interest in open science grows. Simply because progress in science relies on dialogue between scientific peers. The exchange of views between reviewers and authors does not belong behind closed doors.

A version of this article first appeared in Dutch in the NRC Handelsblad, October 26, 2019.

Professor of epidemiology | Emory University, Atlanta USA | Writes about (genetic) prediction, critical thinking, evidence, and lack thereof.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store