Can scientists really be holier-than-thou?

“It’s been four weeks since Sneha submitted her paper and she is in that nail-biting, insomnia-inducing, and anxiously-checking-your-mail-every-few-minutes phase. Only thing one can do at that point is hope the reviews will be impartial and not too harsh!”

All scientists know this gut-wrenching and helpless feeling where you can do nothing but wait when your paper is out for review. Peer reviewing is a bridge every researcher has to cross in order to communicate their findings and its validity to the scientific community. Although it is a bench mark of research quality, it is by no means a flawless concept. It assumes that all scientists are altruistic, moral abiders of the scientific norms. Universalism, one of the four scientific norms, points that scientific validity is free from the sociopolitical status or personal attributes of the individual. But, violation of all forms of norms is a common practice, and scientific norms and scientists are no exception.

Single-blind peer review is the conventional or the most common form of peer reviewing where the reviewers know who the authors are, but the authors are blind to the identity of reviewers. The alternatives to this form of peer reviewing are open or double blind reviews. Although open review process is more transparent, there has been debate that the knowledge of author names and affiliations in both single and open reviews may lead to a more biased assessment of the scientific work.

 

The experiment

To test possible bias in single blind versus double blind peer reviews, Andrew Tomkins and colleagues from the Department of Computer Science, Tsinghua University, Beijing conducted a controlled experiment for papers submitted to the 10th Association for Computing Machinery International Conference (WSDM 2017) which was published in PNAS November 2017 issue [1]. They chose this model as computer science research is first sent to peer reviewed conferences rather than journals. An expert committee was set up which reviewed full-length submissions and four members reviewed each paper. Two of the reviewers had access to author information (single blind), while the other two had none (double blind). Reviewers considered each manuscript and entered a bid (yes, no, or maybe) on whether they were willing to review the paper. They then scored the manuscript ranging from +6 (strongly recommended to accept) to -6 (strongly recommended to reject). The reviewers also provided a ‘rank’ which ranged from 4 (top paper seen by the reviewer) to 1 (the bottom 50% of the manuscripts seen by the reviewer in their batch). Subsequently, the authors collected information as to how single and blind reviewers differed in bidding, reviewing, and entering scores for each paper.

 

The authors wanted to test three particular forms of biases: the Matilda effect, where the papers from first author male papers are given higher scientific credit; the Mathew effect, where famous authors get more recognition; and the third, where biases may emerge due to the acclaim of affiliated institutes. To test these biases, they selected the following covariates for the analysis: female author, famous author, paper form a top university, paper from top company. Tomkins et al. found that the odds were higher for a single blind reviewer to give a positive score to a paper if was from a famous author (odds: 1.6), famous university (odds: 1.58), or a top company (odds: 2.1). They also found that single blind reviewers bid less frequently (22% less than double blind reviewers) and preferentially bid for papers from top universities and top companies. Although they did not find that single or double blind reviewers treated papers from first female author papers differently in this paper, they also conducted a meta-analysis where they analysed their study along with other studies on the effect of gender on reviewing, and there they found significant results.

 

Thus for the same paper, a reviewer who knows that it is from a reputed author or university has a higher likelihood of recommending it than a reviewer who does not have access to this information. Also, if a paper is from a reputed author, university, or company, more single blind reviewers bid to review it and thus, it may be assigned to more knowledgeable or field appropriate reviewers which may further alter the chances of acceptance or rejection of a paper.

 

If the single blind review process seems so murky, is double blind the way to go? If only it were so simple.

 

Wading the tricky waters of double blind reviews

In 2009, an international and cross-disciplinary survey was conducted across 4000 researchers where majority of the researchers (76%) said that they considered double blind peer review “the most effective form of peer review”. But double blind review process has its pitfalls. In most cases, the reviewer can guess the author affiliations if it is from a well-known group. Some journals recommend removing phrases such as ‘we previously showed’, but it is hard to be anonymous in today’s globally connected world as most research groups discuss published and unpublished work in ongoing conferences. Also, most of the papers in physics, math, and computer science are pre-published in e print archive arXiv which makes the anonymity impossible.

Since June 2013, Nature Geoscience and Nature Climate Change have been offering double blind reviews as an option and later it expanded the option to all its journals. Recently, Nature Publishing Group presented the analysis on double blind peer reviewing using data from 25 Nature-branded journals from March 2015 to February 2017 in the Eighth International Congress on Peer Review (London). Out of 106,373 submissions, only 14% of authors opted for double blind reviews in Nature, 12% opted for it in sister Nature journals, and 9% opted for it in Nature Communications. Of those opting for double blind reviews, 32% were Indian authors, 22% were Chinese, 8% from France, and 7% from USA. This suggests the double blind route is preferred by scientists who fear potential discrimination. Ironically, only 25% of papers submitted for double-blind review were accepted compared to 44% submitted for single-blind review.

 

In search for a fair scientific community

Even though studies suggest biases in single blind peer review process, it is still regarded with skepticism amongst the researchers with only one in eight going for double blind reviews.  Also, it has a few kinks in the ‘double blindedness’ with the existence of preprint archives and conferences.

 

So is there any point in double blind reviews?

 

Universalism states that scientific validity should be free from the sociopolitical status or personal attributes of the individual. But scientists, like all humans, harbor conscious and subconscious biases, and double blind peer reviews maybe a small effort, if only in principle, remove them.

 

References:

[1]      A. Tomkins, M. Zhang, and W. D. Heavlin, “Reviewer bias in single- versus double-blind peer review.,” Proc. Natl. Acad. Sci. U. S. A., vol. 114, no. 48, pp. 12708–12713, Nov. 2017.

 

P. Surat Saravanan has a PhD from Tata Institute of Fundamental Research (TIFR), Mumbai and is currently, a freelance science editor and writer.