Is Blind Review Threatened by Online Works in Progress?

Many scholars now post drafts of articles they are writing online before they are accepted for publication. If a submitted article is going through a process of blind review, very often all that a reviewer has to do to to spoil the blindness of the review is to Google the title of the article. Do you think that the proliferation of posting works in progress online threatens the integrity of the blind review process? If so, does anyone have any suggestions about what to do? Simply telling reviewers (perhaps in a stern tone of voice) that they shouldn’t Google article titles does not seem to be a promising solution. And, although a single author who is concerned about having her paper reviewed blindly can simply avoid posting it online before acceptance, this too has its costs. Many scholars want to make their work available for discussion and comment before being accepted so that the article can be improved and so that their work can become part of an ongoing debate as soon as possible. Any ideas?


Comments

Is Blind Review Threatened by Online Works in Progress? — 30 Comments

  1. I think that this is indeed a well-spotted problem since it seems to endanger the “blindness” of blind review processes itself, at least in the referee–>author direction. If the danger is in fact big enough, that is, if it is a vital threat to the whole current practice, then maybe one should really prohibit online-prepublication of a paper unless it is actually accepted by some journal. This need not come at a too high price if the journals commit themselves to speed up the review process in turn (which, probably, they won’t).

    One could also demand of authors that they post their papers only in a non-searchable form, i.e. as pure picture documents (and, of course, also without the actual title under which it is under review). One could back up this option by sending the papers to their referees without any title information at all.

    Yet, maybe philosophers need not “reinvent the wheel” with regard to this issue. There ought to be some already institutionalized solution to this problem in other disciplines, especially the one’s with a far richer and stronger tradition of blind reviewed journals (e.g. physics, biology, economics). So, maybe we should look there first.

  2. If this is really a problem (and it’s not obvious to me that it is – I’m not sure what the motivations are for a reviewer to google an article they’re reviewing) then any solution should be mandatory, like removing one’s name from the manuscript. Otherwise, big names can leave their stuff in the open on google, since knowing the author increases likelihood of publication, while the people that have more need of publication won’t.

    But is this problem really any worse than the already-existing problem whereby people know who’s working in their own particular sub-area, and in some cases have even been at a conference where the paper was presented?

  3. This is a huge problem. Sometimes it is unavoidable. For example, your paper might get accepted for presentation at a conference while it is still under review. And the organizers of the conference may require that the paper (or paper abstract) appear on their webpage prior to the conference (and prior to the journal decision). But in other cases it is avoidable. With a few exceptions I haven’t posted any of my unpublished papers online prior to acceptance. The journal decisions I receive tend to be more favorable when I do not post. This suggests that reviewers do google paper titles, and that various irrelevant factors may influence their decision. I strongly recommend that people do not put their papers on-line until it has been accepted. I make exceptions for book manuscripts, however. Book manuscripts are rarely blindly refereed anyway.

  4. I think Kenny Easwaran is right in mentioning that a similar problem already arises with papers that where previously presented at a conference. Even if outside the referee’s sub-area, conference programs are easy to google for all. Obviously, conference talks cannot be forbidden in the sense that online publication could be (not that I think this would be a good idea).

    An easy solution is then to use different titles for conference papers and the journal paper based on it. At least, this would reduce the chance of finding out who the author is by a simple google search on the title, even if the paper put online is otherwise identical to the submitted version.

  5. Within computer science it is common for journal papers to be extended versions of a previously published conference paper, so double-blind reviewing is often difficult. However, there are other conventions in place that help to provide for a reasonable review. One is that most journals will base decision on multiple referee reports that are shared with authors. Another is timeliness.

    There are only a handful of philosophy journals that follow current scientific editorial practices, however. Synthese is now one of them, which bears mentioning here given its focus on epistemology.

    I think more transparency would be a good thing for philosophy journals. So, it would be nice to collectively reward Synthese, for instance, or to complain periodically about the standards gap in editorial practices between philosophy journals and science journals. While transparancy is not a panacea, it does give you a better idea of what is going on…with your paper, and at a journal.

  6. For those that suggest that junior scholars (in particular) not post drafts-in-progress online because of the possibility of compromising the bling-review process and, potentially, harming one’s chance of publication, do you also suggest not listing ‘works in progress’ on public CVs? It seems to me that this too would have the potential to undermine the blind-review process.

  7. As a reviewer, I haven’t seen a paper I liked in a long time where I couldn’t figure out who the author was. I suspect that papers that have not been vetted at conferences, workshops, reading groups and the like, all of whom have online presences of course, simply do not tend to be up to snuff. In any case, in semantics, which is an almost vanishingly small discipline, it is unusual for me not to know who is working on what anyway. Yes, so long blind review, but I think that on balance the new advantages of openness of science in the internet age outweigh the negatives.

  8. Is it too much to ask that reviewers behave ethically and not look up the author on the web when reviewing? I mean, it’s impossible to prevent them doing that, or to prevent authors putting work-in-progress and preprints online, but surely we ought to be able to ask that reviewers not give into temptation if they are acting professionally?

    Many papers indicate, through the cited references, who the author is anyway – that’s not new. I can usually pick if the author is well published who it is. So why is this any more of a problem now than it was before?

  9. The issue, it seems to me, is not one that ought to focus on the integrity of the author, but rather on the (lack of) integrity on the part of the reviewers.
    Is there a way to check whether a reviewer is checking the material they are to review with the use of the www (eg., via google)? If not, perhaps there should be a procedure in place to maintain the integrity of the review.
    What rules are there in place by the journals for checking the integrity of its reviewers, and if found wanting, what punishments can they give?

  10. Horvath is right in saying that is highly recomended to look out what the hard-sciences are doing with regard to on-line publicize papers and the threat that arise for blind review. But the difference between the hard-sciences and the humanities, is that in the hard-sciences the results and conlucions obtained are reproducible by other´s labs or gropus of investigation across the world, because they have stablished standars and protocols that are objective and very realible in nature, and in essence (in many cases but not, for example, in clinical trials for drugs testing and several other instances) doesn´t really matter if the author is known or ubestknown for the reviewers, just only if the paper pass the require protocols and procedures and is reproducible to be published.

    One posible solution could be what in biology is called: “publication biases” (Song F. et al., 2001, “Publication and related biases”. HEALTH AND TECHNOLOGY ASSESMENT. 4:1-115; or Moller A. and Jennions M., 2001, “Testing and adjusting for publication bias”. TRENDS IN ECOLOGY AND EVOLUTION, 16:580-586.) in which is made a comparison and a meta-analysis between published
    (publicize)papers and potentially but not published (publicize) papers to see the harm done. The problem is that both papers must have similar conclusions and this is not ussually the case in philosophy.

    Onother possible solution is to create a large-scale anonymous philosophical “bankdata” organized by area and date of submission of papers with only the body of the text for visitors to ammend or disscuss it. The author/s of the paper are randomly assigned with a number to be identified thereafter.

  11. I find it hard to believe that in cases where a paper is written in a sub-area, and the editor picks a person in the sub-area that keeps up with the literature and trends in the sub-area does not have at least a good idea who wrote the paper they are reviewing. In order to be a qualified reviewer in certain cases one should be a competent reviewer. “Competence” should include knowing the trends and the major movements. The comments above suggest that I am not alone in thinking this. Given that papers are presened at conferences, footnotes often list people the person has talked to, examples in the work are examples from prior work, and writing style gives away training it is hard to believe that blind review is “I have no idea who is writing this paper” reviewing. I also find it hard to believe that people reviewing don’t as a knee-jerk reaction wonder who wrote the paper. And if they don’t wonder who exactly wrote the paper, they at least wonder from what approach or tradition of the sub-group the paper is being produced from. I don’t think, as others point out, that posting information about papers in progress produces anymore of a problem than the other blind-review defeaters. The main thing, which is philosophically interesting, is whether a person can give an objective report even when they know who the person is, or from which sub-group the paper is from. Clearly knowledge of who wrote a paper does not imply that the judgement is not good. In addition, It is the reviewers responsiblity as a professional to give an honest appraisal of a paper. Note that other things appear to be problemantic. It is probably the case that on occasion even when a person does not know who wrote the paper they do a bad job anyway because they are lazy. Many people complain about getting comments that are bad or do not make sense. This is just another violation of the reviewers responsbility. I raise the following question for discussion. Consider the following situation. Person B always writes a paper with example p in it, and is the only person that writes papers with example p. Person A is given person B’s paper for blind review. Person A has never met Person B. The paper has no footnotes, has never been presented at a conference that A has attended, but A has read a paper by B before, and is aware of the fact that only B uses example p. Should A report to the editor that they can no longer review the paper?

  12. A couple of comments have suggested that it is unlikely that a referee will be unaware of the identity of an author when the referee is well-chosen — that is, a specialist in the field in which the paper is written. But this assumes that the author of the paper is an established and well known philosopher. One of the motivations for blind review, I take it, is to protect precisely the unestablished. The problem, then, is not simply that we are able to find out the identities of the established authors of the papers we referee. Rather, it’s that we are able to find out that the authors of the papers we referee are not established — either because their papers are posted on their websites (and who on earth is THIS person?)or because their papers are not posted anywhere (that is, on the websites of established philosophers or on conference websites). Of course, ideally we wouldn’t let this knowledge impact our reports. But, then, if we were so good at ignoring such knowledge, blind refereeing wouldn’t be necessary in the first place.

  13. I like Kai von Fintel’s remark about the net advantages of openness of science. But whether or not you agree, this trend will only continue to accelerate. Some of the suggestions in this thread for thwarting online searches are, or will soon be, very easy to work around. We all live in glass houses now.

    Perhaps then the problem should be to see how to effect fair vetting of manuscripts in an open environment. Insisting on reports is one component. Another is spelling out why this is important to the profession, and not simply important to authors. We all want a quick and complete mechanism for communicating contributions to our field(s), after all.

  14. Rather, it’s that we are able to find out that the authors of the papers we referee are not established — either because their papers are posted on their websites (and who on earth is THIS person?)or because their papers are not posted anywhere (that is, on the websites of established philosophers or on conference websites).

    I guess the question is not whether they’ll be able to find the paper, but whether they should be looking for it. My guess is that–and this came up on a related thread–most papers are Googled long before “blindly” refereed. So if it’s been posted anywhere, or discussed on some blog, or presented somewhere, in some conference, chances are it won’t be blind reviewed. Might be a good idea not to submit the paper under the same title.

  15. Pingback: phonoloblog»Blog Archive » The future of blind review in an online world

  16. Something worth noting is that changing the title of a paper won’t keep it from Google’s watchful eyes. Someone need only search for a some peculiar *phrase* in the paper to find an online draft… and while changing the title for submission is simple, a complete rewrite to eliminate these peculiar phrases is not.

  17. Kai von Fintel said “As a reviewer, I haven’t seen a paper I liked in a long time where I couldn’t figure out who the author was”.

    This suggests another problem. Some reviewers might choose not to like a paper because they can’t figure out who the author is (I am of course not suggesting that von Fintel is doing this, just that some might). I think that it is definitely true that if you know that a paper was written by a well-known philosopher or linguist, you will be more likely to accept it (I know I am).

    Yes, papers should be “vetted” at conferences, reading groups etc. before they get published. But I doubt that someone working in, say, ethics is aware of all the reading groups taking place in their area of specialization. And what about ethics conferences (including graduate student conferences) in Australia, England, Germany, … We can’t go to them all. And I doubt anyone working in a reasonabley broad area of philosophy can keep up with them all.

    To guarantee a fair evaluation process I think one should keep one’s paper off one’s website until it has been accepted for publication (the exception being papers that have already been presented at conferences). As for works in progress. I think students and young philosophers (or linguists) should just list paper titles that are reasonably close to the original on their CV.

    It is true, of course, that there are many ways in which the objectivity of the evaluation process can be compromised. But that doesn’t mean that we shouldn’t try to make the process as objective as possible. Of course, we should. If a paper is good, it is good (regardless of how well-known its author is).

  18. I think that it is clear that we all want the process to be as objective as possible. I do think, as Brit gestures at, I was hasty in implying that in all areas one who is competent would be able to determine the author or roughly who the author is. Ethics is a good example of where it would be hard, and semantics perhaps a good one where it would be easy. And, as Jermey points out, I may have assumed something about estabilshed vs. non-established authors. Blind review should protect those that are non-established. However, regarding the specific issue raised perhaps the central point is what John Wilkins points out — those that are asked to review a paper must recognize that names are not important, and simply not pursue searching out who the author is, though it is possible to do so. I think part of the other worry that is floating around is that there appear to be other blind-review undermining features that are in papers, and there are many techniques for finding a paper even if the title is changed. There is no way to get rid of them all, for all cases of reviewing. Ultimately, it might be best that an author upon submitting a paper for review take the paper down from web access, and remove mention of it from other web accessible items, such as a C.V.. This would at least allow the paper to be circulated and discussed prior to submission, and not be available during the period of review.

  19. I’d like to reiterate John Wilkins’ point. Why isn’t this the most plausible thing to say about the issue: the practice of posting papers online, and the fact that everyone has access to Google, has made it easier than it was to do something unethical, namely, corrupt the blind-review process by finding out who the author of the paper you’re reviewing is. As usual, when technology makes being unethical easier, the solution is not to blame the technology, but to blame the people who are behaving unethically.

    So there might be no way to regulate and control this unethical behavior. If that’s true, then the solution to the “problem” is to add “Googling blind-review paper titles” to the list of unethical behaviors that we can’t regulate or control.

    Jeremy Fantl is exactly right, I think, in saying that this is an issue that effects unknown junior philosophers, not established hotshots. Relevant here is not merely the identity of the author, but the identity of her institution: unknown junior faculty usually don’t have bad reputations, but schools that don’t do well on the Leiter report, for example, do.

    I am relatively new to the reviewing game (I’ve reviewed 3 papers), but I have managed to “resist the temptation” to use Google to discover the identity of their authors. (I checked up on them after I submitted my reports, of course!) I guess it might explain some things about the world if I turned out to be some kind of moral saint, but I don’t think it’s too much to demand of our colleages that they possess the minute scrap of moral backbone it takes to “resist the urge” to violate the blind-review process.

    (I am not sure, by the way, where this “temptation” comes from – typically we are tempted to do something unethical when it will be in our own best interest, right? How does it benefit the reviewer to know the identity of the author, in these cases? There’s curiosity, to be sure, but is satisfying this curiosity such a delight that it’s worth throwing your integrity in the garbage? Compared to the immoral pleasures out there, surely the thrill of knowing whose philosophy paper you’re reviewing is relatively unappealing.)

    I have also never been able to deduce an author’s identity just by reading a paper. Maybe that means I’m not a “qualified reviewer,” but I have a feeling I’m not alone on this.

  20. “typically we are tempted to do something unethical when it will be in our own best interest, right? How does it benefit the reviewer to know the identity of the author, in these cases”

    You can potentially review the paper much more quickly (according to the principle “hotshot –> yes, no-name –> no”)

    I don’t walk on the street after 8 p.m. I don’t just note that unethical acts are unvoidable.

    As mentioned earlier, my reports from reviewers have been far more favorable when I have kept my papers off my site.

  21. There’s curiosity, to be sure, but is satisfying this curiosity such a delight that it’s worth throwing your integrity in the garbage? Compared to the immoral pleasures out there, surely the thrill of knowing whose philosophy paper you’re reviewing is relatively unappealing.

    It’s a good question. It probably has something to do with regarding these invasions as more or less on the level of getting gossip. I wouldn’t underestimate the appetite for it.

  22. Many thanks for the great discussion of the question I posted. Although it is not clear how the institutions of our profession should respond to this problem, I’ve received a very clear answer on how I should respond: I have taken all of my articles that are works in progress or under review off of my website. This certainly seems to be the prudent route for a young scholar like myself.

  23. I guess this has been noted in the above discussion, but just to reiterate: online publication of papers shouldn’t really be the focus. The question is: are you going to put up online any information that’d allow a google search to link you with a submitted paper? E.g. lists of work in progress on your website, lists of work in progress on an online CV, etc. Plus all those other events that associate the paper with your name: e.g. conference and research seminar presentations of it.

    Personally (as a junior philosopher) I’ve found many benefits in putting papers up, of which getting comments back from people in various forums was a huge thing. So I continue to put stuff up, so long as I think it’s in a fit state.

    I do think the idea of changing a title when submitting a paper is a nice suggestion: relatively cost-free way of making the identification harder.

  24. I think Jeremy is exactly right that anonymous refereeing protects the younger and less well-established people more than those who are better known. And I think this means that we don’t need any sort of mandatory system requiring people not to post their papers. People can keep their own work off the web if they like and protect themselves from a risk of bias if they so choose and it will be effective because of the way that uncertainty keeps things honest. I’ll try to explain what I mean below by using my own refereeing experience to illustrate.

    I find it keeps me honest when I’m unable to tell whether the paper I might reject is written by someone whose philosophical skills I admire (hence I’ve resisted Googling papers I referee). So long as I don’t know that it is not someone I respect, I have an extra reason to carefully read what I’m refereeing. Thus people who want to protect themselves by keeping their work anonymous can do so by keeping them off the web, as Brit does. You just cannot infer from the fact that you can’t Google a paper that it is not written by a well-known and smart philosopher. If anything the most well-established are less likely to have the time or interest to put papers up on the web, though of course many do. So keeping one’s own papers off the web is one way to create the desired uncertainty if it worries you. (Myself, I prefer taking my chances to get useful feedback, but I’ve got tenure and am not taking much of a risk.)

    The better known will still have an advantage, but not especially because papers show up on the web. I actually doubt that the presence of papers on the web is the main way in which authors are identified. Internal evidence and the background knowledge of referees plays a pretty large role in unmasking authors. People really should take more care to keep references from which they can be identified out of papers they submit. I have gotten annoyed when refereeing a paper that has obviously not been suitably edited to remove identifying references because I think it gives the better known an unfair advantage. You can replace references to people who helped you with ‘omitted to preserve anonymity’, and you can also omit references to your own work or reword them so as to make it unclear you are citing yourself. My experience is that only about 60% of the papers I get do this sufficiently well. Some journals (such as Ethics which razors out obvious self-references)do a better job than others policing these things.

    Still, of course, sometimes it will be impossible to keep the author hidden. Papers filling in or taking off from a person’s well-known project will be easy to identify. And with enough background knowledge one can often take a pretty good guess about who wrote something. So some bias in favor of the well-known may be hard to avoid, and you can’t blame these people for that so long as they make the requisite effort to remove self-references.

    I think this adds up to the thought that a combination of referees resisting temptation to Google, authors keeping papers off the web if they think that lack of anonymity will hurt them, and better editorial enforcement of anonymizing papers will be the most realistic but not perfect solution to any problem that arises.

  25. “I have gotten annoyed when refereeing a paper that has obviously not been suitably edited to remove identifying references because I think it gives the better known an unfair advantage.”

    Yes, me too. I see this ever so often. I find it really odd that some authors apparently do not want their work to be evaluated on as neutral grounds as possible.

    I should also mention that Brian Weatherson has commented on this post on his blog Thoughts Arguments and Rants (in case you haven’t seen it). He makes some interesting points that are worth taking into account, especially, if you are on the job market. Your best work will most likely be your most recent work (not always but in many cases). If you keep it off your site it won’t be available for search committees to read. In such cases you might want to risk posting (though, personally, I still wouldn’t want to jeopordize the objectivity of the evaluation process).

  26. I don’t know how it is with philosophy, but in my field many people post drafts online in order to claim ownership of ideas and to discourage referees from rejecting your paper*in order to* steal your ideas. Many junior students have wonderful ideas but poor presentation skills, which makes them vulnerable to being exploited by referees who reject their paper on stylistic grounds and then write a paper of their own without any acknowledgement to the poor students.

  27. Why not have an “honor code” for reviewers, analogous to the honor codes many private colleges have. To be a reviewer at a given journal, the journal makes you sign something saying you won’t do various things, Googling to find out who an author is is one of the things that will be explicitly forbidden. Sure, nothing will be preventing the reviewers from breaking the code, but my guess is that having them sign such a thing actually would reduce the rate of the objectionable behavior. Having been at schools with and without honor codes, I definitely have the impression that having students sign an honor code really does reduce the rate of cheating. And it would be such a simple thing for a journal to do.

  28. Pingback: Footnotes on Epicycles

  29. Pingback: S&P: Editors’ Blog » Blog Archive » Stalking the Perfect Journal

Leave a Reply

Your email address will not be published. Required fields are marked *