"Peer Review"

HELEN

Very often on the creation-evolution forums, evolutionists will challenge the creationists to produce evidence and references from peer reviewed material. What 'peer reviewed' means is that an article submitted to a journal or other publication is then sent by the editor to two or more people who are considered reliable experts in the particular field the article is dealing with. These reviewers then check the article for accuracy, consistency, and such and return the article with their review to the editor. The article may be considered ready for publishing immediately, in need of some minor 'repairs', in need of some major adjustments and a rewrite, or simply unsuitable altogether for that journal for any one of a number of reasons. The editor then decides what he wants to do based on the opinions of the reviewers.

Although authors are sometimes asked to suggest reviewers for their articles, the editors of the major journals already have a list of reviewers they use for the different areas of research their particular journals cover.

At its best, this system weeds out junk material and makes sure that the material published is of the best possible quality and workmanship.

At its worst, however, the peer review system only perpetuates the ideas of the editors and their chosen peer reviewers regardless of conflicting data or the sloppiness of accepted articles.

Typical of all of us, the peer review system includes both the best and the worst. An article recently dealing with this subject quotes a critic as saying, 'Not only did peer review pander to egos and give researchers licence to knife each other in the back with impunity, he said, but it was also "completely useless at detecting research fraud" and let editors off the hook for publishing poor quality studies.'
( http://bmj.com/cgi/content/full/326/7383/241/a )

In my own personal experience as both a peer reviewer and a pre-editor who helps prepare articles (at the request of the authors) for submission to a journal, I have seen some horrid abuses of the system.

1. I have seen a peer reviewer refuse to submit his critique for an extended period of time, effectively blocking publishing of an article he didn't agree with but couldn't really find fault with.

2. I have seen peer reviewers refuse to review the work of an author they didn't like personally. (Although the articles generally go to the reviewers with the author's name 'unknown', the work of a number of well-published individuals is well-known enough for the reviewer to have a pretty good idea whose work he is reviewing.)

3. I have seen peer reviewers pass articles that were later proved to be incorrect, being too lazy or rushed or both to check the references in the article to make sure they were being used correctly.

Because the editors themselves can pick the peer reviewers they want, the views of the editors are those promoted by the journal. This happens both on the evolution and creation sides of the fence. Thus, when an evolutionist challenges a creationist in a forum to please produce evidence from a 'mainstream' (meaning evolutionary) peer reviewed journal, what he is really asking for is some kind of evidence against evolution being published by a journal which is promoting evolution. It is only by accident and a great deal of in depth research on the part of the creationist that this can be done. There are snippets, which are often not peer-reviewed, in news items and such in these journals which can be gathered indicating that an iceberg of information lies behind what is not being said. But it can be hard. Creation journals are, of course, considered nonsensical by evolutionists.

The fact is that both sides are guilty of the same thing: not maintaining quality but favoring only those articles promoting their world-view.

So, for creationists who receive this challenge, I would offer the following: don't worry about it. Look for the truth. God's creation does not contradict His Word. Don't be content with the commonly publicized material from either side. Be willing to dig a little.

For instance, one thing I am very curious about now is the report of a 'four-winged dinosaur' series of fossils coming out of China. Same place which forged another fossil of a dino/bird transitional a couple of years ago. I'm half expecting a 'made in China' sticker to be attached one of these days. Yet, despite the bizarre claims being attached to this fossil, the prestigious journal Nature published a full article proclaiming the importance of this find.

I question just how accurate a lot of things are now when they proclaim something astounding having been discovered.

Peer review means almost nothing in terms of accuracy anymore. No matter where you read something, I would urge those interested to dig for the truth of the matter anyway.

 

MDKLUGE

Helen has written complaining about alleged faults of the scientific peer-review system, particularly regarding articles submitted by creationists. Below I shall explain why I think her criticisms of the peer-review process are largely without merit. It should go without saying that I do not suggest here that the peer review process is perfect or that it could not be improved. Here I confine myself to Helen’s complaints.

We start out with a minor inaccuracy:

Actually many journals (e.g., Physical Review A-E ordinarily use only one reviewer (or referee). If an author is having trouble getting something published in a Journal using two or more referees, especially if there is substantial disagreement between those referees, it might be worthwhile to try to get published in a Journal using only one referee. The more referees, the more likely that someone won’t like something.

That is true, but misses the point. Although Editors are not bound by authors’ suggestions for reviewers, those suggestions are important (if not the most important) source material for Editors’ lists of prospective referees. Suppose, for example, that I submit a paper “On the Smith Effect in Whiffnium doped with Whaffnium at Temperatures between 1 K and 10 K”. Suppose further that I recommend Jones and Brown as potential referees. The Editor looks at my paper’s title (and also at whatever classification number I’ve submitted indicating the subject of the paper) and basically looks for key words like “Smith Effect”, “whiffnium” or “whaffnium”. The Editor looks in her list of potential referees to see if any of them match one or more key words. If she finds Jones or Brown among the matches she may ask one of them to be a reviewer, or she may select someone else from the list who matches. But even if neither Brown nor Jones is selected as a referee for my paper, the Editor has at least been alerted that Brown or Jones may be considered as reviewers of other similar papers. Or perhaps the Editor couldn’t find a close match in her list. Then she might be inclined to ask Jones or Brown to review my work, and to add them to her list. Editors’ requesting names of potential referees is not intended to be decisive in referee selection in any instant case. It is, rather, a way for Editors to become familiar with the views of potential authors on who is expert in a field.

It is hard to see how. There are hundreds of Journals, and for papers on most subjects literally dozens of Journals suitable for publication. (Of course modifications in paper style would have to be made to accommodate requirements of each Journal.) If the Editor of some Journal is pig-headed and systematically blocks publication in her Journal of articles with which she disagrees, then authors will go to other Journals. It’s not that there is no abuse by Journal Editors, but that the peer-review system itself can correct such abuses. For the peer-review system to fail here, essentially all of the Journal Editors in some field would both have to hold the same rigid views on some subject and decline to permit anything contrary to those views be published in their respective Journals.

Such editorial closed-mindedness is not impossible; but is unlikely in the competitive world of scientific publishing where new idea are sought and prized. Of course even among individuals who seek and prize new ideas, prejudice may prevent fair appraisal of some particular new ideas; but in a group as scientifically diverse as is that of Journal Editors, it is unlikely that all would share irrational prejudices against the same new ideas.

That it is unlikely is not to say that it is impossible, but for any particular new idea rejected by the consensus of Journal Editors of Journals from relevant scientific fields, it is much more likely that the new idea is just unsound than to believe that all Editors are prejudiced. Thus it must be considered a significant difficulty if no Journal, or no Journal Editor can be found willing to publish something. (It is of little interest if some particular Journal or Editor refuses to publish some particular new idea.) In the case of creationist material failing or not making scientific Journal peer review virtually everywhere, that failure should be of great concern to creationists, and should strongly suggest that there is something seriously wrong with their ideas. It isn’t proof, but it should be cause for creationist introspection. Instead we hear complaints about the peer-review process.

One would expect in Helen’s article above, critical of the scientific peer-review process that she would present some evidence that that process is flawed. Instead she presents someone’s testimony that it is flawed! It is true that the quote is from a study on the effectiveness of peer-review; but the study is quite off point.

The study concerns peer-review in the biomedical field rather than some ordinary branch of science like physics, chemistry, biochemistry, biology, geology, astronomy, etc.) Biomedical research is virtually unique in its goals; it must inform both researchers and clinical physicians (who are often inexpert in the fields of articles they must consult). Unlike other branches of science, detecting quackery or other fraud is of paramount concern. In most branches of science peer review is not intended to be particularly effective against fraud.

(Indeed, few effective measures against scientific fraud are in place. The scientific community relies largely upon trust to prevent fraud. The wisdom of that reliance may be questioned, but as scientific peer review is our subject here, and as scientific peer review is not meant to suppress fraud, criticism of a possible excessive trust within the scientific community belongs in a different discussion.)

As for the study authors’ contention that (biomedical) peer review gives license for reviewers to “knife each other in the back with impunity”, that is a subjective conclusion, a dictum by the study authors to be understood as commentary on their study’s conclusions, and must not be concluded with the study’s conclusions themselves. Would their comment translate to non-biomedical disciplines? Perhaps so. Almost everyone in science can provide anecdotes of reviewers being overly harsh or sarcastic in their reviews. It may be a real problem with the scientific peer-review process; but even so it is peripheral to the problem alleged by Helen, the scientific peer-review process’ failure to permit publication of meritorious creationist research in scientific Journals. If Helen could show that creationist authors were “knifed in the back” more frequently or more deeply than non-creationist authors of works of similar subject and quality, then the “knifing” matter would be relevant here. As it is, however, it is just bait and switch. Helen alleges one problem with peer-review and quote-mines about another.

In any case, from Helen’s remarks one might easily have missed the study’s real conclusion: That existing studies do not permit one to conclude much of anything about the effectiveness of scientific peer-review (in biomedical research). Indeed, the studies reviewed did not even consider the effectiveness of peer review itself, but rather on such aspects of peer-review as keeping referees or authors anonymous or not, of the effects of the medium of submission of review on review quality and the like. The study was most emphatically not about the quality of peer-review itself.

I’m curious: Did the referee send a letter to the Journal Editor saying “I disagree with the paper, so I’m not going to submit a review”? If not, then how can you know the his motivations? Sadly, referees have lives and busy schedules just like the rest of us. This includes referees who happen to disagree with the paper they are currently supposed to be reviewing. When a referee fails to submit a report in timely fashion it is the Journal Editor’s duty to tactfully remind the referee of his omission, or ultimately to select a different referee. From an author’s point of view, delays between Journal submission and publication are frustratingly long. Many dawdling referees would no-doubt benefit from a swift kick to the derrière, but mostly for sloth, not malice. In any case, even if Helen’s tendentious interpretation of this event were correct, it would be an example only of a fault with this instance of peer-review, and not the peer-review process systematically, in general.

That’s what a referee is supposed to do if he believes that he cannot give an author a fair review. Of course this should be done in timely fashion, and the referee is under no obligation to give a reason for declining to review the manuscript. Again, though, even at its worst this would be an instance of a personal abuse of the peer-review system.

A minor correction here: Although some Journals conceal authors’ names from referees, many only conceal referees’ identities from authors. While perhaps frequently “go to the reviewers with the author’s name unknown” that is not generally true.

It is not the purpose of peer review to detect every error. This is no more a “gross abuse” of peer-review than it would be a gross abuse of blood testing procedures if (say) 1% of blood samples contaminated with HIV passed blood-bank screening and the test being used was the best available capable of routine administration. To be sure, efforts would be warranted to improve blood testing procedures, just as the publication of incorrect papers may suggest problems with the peer-review process which need be addressed. Such problems, however, are not soluble by lay discussion. In any case, this is opposite the problem Helen alleges for creationist authors: Even Helen does not allege that their incorrect articles are being passed by scientific peer review.

No, for the reason Helen gives the views of Journal Editors COULD be [restricted to] those of the Editor, but that it actually is so follows no more from the premise than that, because this forum’s Moderator either reviews himself or selects prepublication reviewers for articles submitted for this forum, then the views expressed in articles in this forum must reflect the views of our Moderator.

Perhaps there are Journals promoting evolution in the sense that some Journals “promote” Newton’s Laws of Motion, Faraday’s Law of Induction, or the use of the Schrödinger Equation to solve (non-relativistic) problems in quantum mechanics.

Then do it! All scientific research is tough, requiring work at a great deal of depth. Real scientists don’t complain about it; they publish their deep work in the same Journals from which they read the deep work of their colleagues.

The impetus for this thread, you might recall, is the challenge frequently issued to creationists to show peer-reviewed work (in reputable scientific Journals) supporting them. Helen now says that (like all scientific research) this, “by accident and a great deal of depth…CAN BE DONE. [My capitalization—MDK]”

Well, if it can be done, then please do it! And when you do it, then, please show me the money.

Experienced scientists know that they can rely upon Journal articles themselves far more than upon news reports about them—even news reports within scientific Journals themselves. News items, unlike peer-reviewed articles, are written by journalists, not necessarily scientists, for nonspecialists or even lay people. They are not the best evidence available. If you want to know what is being said, or even what is not being said, you must read and understand the underlying peer-reviewed articles. What you imagine exists under the tip of the putative iceberg whose existence you glean from a news report is just that—something from your imagination. News reports, even scientific news reports, are inherently incomplete descriptions. Sometimes they are reliable sources of what was said; but they are unsuitable for guessing what might have been left unsaid.

That’s fine as long as creationists don’t pretend that creationism is in any way scientific or that scientific evidence supports it. If creationists would quit pretending that creationism was science, then no one would ever issue the “challenge” to find something supporting creationism in the peer-reviewed scientific literature. It wouldn’t make any sense. The fact is, though, that creationists claim their creationism to be “scientific.” It goes without saying that such a claim by creationists needs support from the scientific literature in the form of their idea’s presence there. Otherwise creationism, whatever else it may be, is invisible to the scientific community and is no part of science.

That is the first part of the “challenge”. Look deeply in the peer-reviewed scientific literature for something pointing to creationism. THEN TELL US ABOUT IT!

Maybe so, but off topic. Helen presents no evidence that, even if publication was erroneous, that putatively erroneous publication was the result of faulty peer-review. That a broadly similar fossil from nearby was recently reported would be suspicious, but hardly dispositive In any case, the “challenge” concerns correct putlished creationist research, not incorrect research.

Certainly everyone here will agree with Helen’s last sentence. Unfortunately Helen’s claim that “[p]eer review means almost nothing in terms of accuracy anymore” reflects her almost complete misunderstanding of what scientific peer-review is supposed to do. She might arguably be correct had she referred only the accuracy of actually published material. Indeed, a manuscript’s passing of peer-review and subsequent Journal publication is not, and has never been intended to be, a Good Housekeeping Seal of Approval for the article in question. The virtue of scientific peer review’s effects on accuracy lies primarily in what scientific peer review prevents from being published (and the obvious defects which it spots in articles eventually published). If Helen wishes to show that the peer-review process has unjustly harmed creationists (either in their own manuscripts or those of others tending to support creation) then let her show how this is systematically true. So far I haven’t seen even one example of a creationist writing submitted to a scientific, peer-reviewed Journal inappropriately denied publication because of unfavorable peer review. Absent a systematic inappropriate denial of publication of creationist material in scientific peer-reviewed Journals, the “challenge” remains, and creationists cannot blame peer review for their continued failure to meet it.

 

HELEN

Hello, Mark! Long time no see!

That’s a long essay for saying a few things. I think you missed the point of what I was saying originally, however. It was not actually a complaint about creationists getting published – it was rather something to help those who are on creation/evolution forums when they are asked to present support from mainstream peer-reviewed journals. I wanted them to know what was really happening.

First, sorry about the minor inaccuracy of sometimes only having one reviewer check an article. Yes, of course that is true.

However your many words did not cover up the fact that editors, especially of the larger journals such as Nature, Science, and JAMA, have their own lists of preferred reviewers whom they know support the general stand of the journal. This does ensure that the stand of the editorial staff of the journal will be upheld. That was my point.

Nor does it matter that there are many, many journals. That is essentially beside the point. As I pointed out, even the creation journals have the same problem/policy: the stand of the editors is going to be upheld and new material and challenges will be tolerated up to a certain line and no further. That’s simply the way it is. This was not even a complaint it was simply a matter of information for those who participate on forums and get challenged regarding support for what they say.

The one thing you did say which I found rather interesting was “In most branches of science peer review is not intended to be particularly effective against fraud.” And this is a very important point for people to understand. Just because something appears in a peer-reviewed journal does not guarantee its trustworthiness. This is another reason creationists in particular do not need to allow themselves to be intimidated by demands from opponents regarding finding support out of a peer reviewed journal which the challenger usually says he or she has to approve of. That can be waste of time, especially if actual data is available outside that venue.

You are also right when you write that “few effective measures against scientific fraud are in place.” You claim the science community relies largely upon trust to prevent fraud. If that is actually true, then the scientific community as a whole is absurdly naïve. Where this applies to peer review is that the reviewer will tend to trust the person if the person is famous in his field! And a person who is famous in his field has a LOT to lose if his work does not pan out. This then becomes a combination of both self-defense and pride, neither of which are known to be primary promoters of honesty.

So I do thank you for adding to the warnings I was giving the folks here. They are timely.

Now, quickly, to respond to the questions you asked about the examples I brought up:

1. Yes, the editor did prod the reviewer. The reviewer in this case responded with a series of promises about when the article would be ready, none of which were kept. The article was eventually withdrawn by the author (I presume in disgust – I don’t know all the details). Nor do I know how often this occurs. But evidently it IS one way to either delay or prevent a paper from being published, and that is something I was not even aware of happening before this incident. I have been told that it is not that uncommon, especially if the reviewer is aware of another paper coming to print which he prefers and wants to have a head start.

2. In my own opinion, personal dislike should not get in the way of science or peer review, but that is perhaps me being naïve now.

3. Peer review is supposed to catch sloppy work. It is supposed to be done by folks who are basically aware of the literature in the field and can tell if the references have at least been read by the author(s). As other studies have shown, very often the authors have not bothered reading a good number of the references they cite.

I have seen excellently written peer reviews and I have seen ridiculous ones with the different authors I have worked with. Both exist. The good ones had enough in them that I sometimes felt the author should have paid the reviewer for his knowledge and time! I have seen some of what I consider the best in the field. But I am also aware of the fact that the best (by virtue of the definition, I suppose) is not at all the norm, and a lot of slipshod material can sneak through, as well as a lot of material that is accepted simply because it is something that boosts the editorial position.

And again, this is the reason I wanted folks to know what was going on. I was not complaining. I was informing. You really did miss the point of the post.

By the way, this forum is explicitly for the discussion of opposing views and is no way to be compared with a peer reviewed journal. When my posts have been edited, it was for wrong formatting, personal material which did not belong here, and an occasional insult that was sparked by my temper. I have seen the posts of many of the evolutionists who have posted here when they have posted on other forums and I am quite sure the personal insults have been edited out of theirs, too, but I doubt if any of the real information has been.

In closing, here are some links supporting some of what I have been talking about:

http://filebox.vt.edu/artsci/geology/mclean/Dinosaur_Volcano_Extinction/pages/corrupt.html

http://post.queensu.ca/~forsdyke/peerrev4.htm

http://beaugrande.bizland.com/peerreview.htm

http://ethics.ucsd.edu/courses/integrity/assignments/review.html

 

MDKLUGE

Helen wrote:

Helen is correct that I misunderstood the thrust of her argument. My response, above, is therefore largely unresponsive to her main concern. That is not to say that my response was incorrect in any of the points it made to Helen’s individual criticism of the peer-review process. I think my posting was correct and I stand by it, but it does not respond to Helen’s main concern.

I did indeed think that Helen was complaining about the peer review process improperly preventing creationists or creationist material from getting published in scientific Journals. That would have been a valid argument on her part against the challenge. If the premise that creationists are unfairly discriminated against in the peer review process is true, then the challenge based upon the fact that creationist material is seldom published in scientific peer reviewed journals would lose its force. Indeed it is the only valid argument that could be made. The argument that Helen now makes is unresponsive to the challenge.

Helen’s argument boils down to a denigration of the peer-review process and a suggestion that creationists not worry about a challenge that creationist material isn’t published in Journals employing such a flawed process. In this she misunderstands the nature of the challenge. Perhaps her misunderstanding is our fault for issui8ng a challenge to lay creationists in terms understood by scientists. When we issue the challenge it is not per se our concern that creationists’ articles seldom if ever go through the standard scientific peer-review process. When we say “publication in a peer-reviewed scientific Journal” we mean publication of regular articles in what are generally regarded as legitimate scientific Journals. “Peer-reviewed” is added to emphasize that Letters to the Editor and the like don’t count. Although most scientists value the peer-review process more than does Helen, our emphasis in the challenge is not upon peer-review, but upon (lack of) publication of creationist material in the regular and available channels of scientific publication. In other words, the challenge is to show creationist material communicated to scientists through regular scientific communication channels.

Now a moment’s reflection indicates that nothing said about those channels of communication or the peer review process alone can meet the challenge. Something about the relationship between creation research and the regular channels of scientific communication that is peculiar to creation research and not to other areas of research must be found. After all, hundreds of new ideas spawning numerous new areas of scientific research have gotten past Journal Editors having contrary ideas. Even if we grant for the sake of argument Helen’s contention that the peer-review process is as corrupt as she says it is, the challenge is why so many other areas of research survive and even flourish, while scientific creationism not only fails to flourish, but to all appearances hasn’t even had its scientific germination.

Why hasn’t scientific creationism been communicated among scientists in regular channels of scientific communication? Why instead has it been communicated exclusively to the general public? For forty years it has produced nothing in the scientific literature. In that time it hasn’t even produced a credible body of attempts to communicate in the regular (peer reviewed) channels of scientific communication. Even if all of the Editors of all of the scientific Journals were irrationally prejudiced against scientific creationism there is no evidence that creationists have ever seriously sought to change the minds that count. All we see is communication to the general public.

 

HELEN

Mark, I know that you and many other scientists (yes, folks, this Mark is a very competent scientist) depend upon publishing in peer-reviewed journals for peer and professional respect and advancement, and so there is no way you can afford to admit to basic flaws in the system.

However those flaws do exist, and they are major. In fact, those peer-reviewed journals which do NOT promote the current reigning paradigm but which either allow a variety of views or promote another view are considered by mainstream scientists to be either ‘fringe’ or ‘non-science’ or other derogatory terms. Thus no major challenges to the main paradigm will be seen in the mainstream peer-reviewed journals until the point comes when it is impossible to not present them, because they turned out to be right. Wegener and plate tectonics are an excellent example of this. It is too bad he did not live long enough to see his work, so ridiculed and scorned by virtually everyone in the field of geology, suddenly become the true cutting edge of real geology and accepted as the explanation of so much. Who would publish him while he was alive?

Yes, there are some major flaws in the process, as my original post stated.

So if you want some complaints about peer review, I can offer them. But I also know that it is a process that is almost necessary to weed out junk. So many people want to be published that the journals must pick and choose, and peer review, at its best, is an excellent way to do it. I wish it were always at its best, that’s all…

And that has NOTHING to do with creationist materials being published. I am not even intimating that in the above paragraphs. Please understand that. Wegener, for instance, was not a creationist!

But again, my original post was to simply inform creationists that the challenge often posed by evolutionists to them on various forums to reference what they are saying to mainstream peer-reviewed material is a deliberate red herring and straw man. Views distinctly contrary to editorial positions are not published by the journals of these editors – on any side of any fence!

The best thing for anyone who is arguing for any position to do is to check data; do a little homework. Believing ANY popularizer, whether on the evolution side it be Eugenie Scott, Stephen Gould, or Richard Dawkins (which is, essentially, what they all have been regardless of background), or on the creation side it be Kent Hovind, Ken Ham, or any of the others who are not involved in the scientific research, is basically folly. One can read them and even accept a lot of what they present, but without knowing the facts, one is often putting oneself – again on either side – in the seat of the fool.

You, Mark, do basic research. You collect data – the raw stuff – and work with it. My husband does the same, even though you two disagree with each other. I would rather watch you two in a room together hashing out your differences than see any number of lovely video presentations promoting either side.

But that takes energy, even to listen! And most people want their ideas nicely wrapped up in a half hour presentation they can memorize and not understand. I was trying to warn against that, basically. I’m afraid that depending on peer reviewed material, in a number of cases, is as bad as depending on popularizers and videos. And so I encouraged the folks here to dig for themselves if they really wanted to enter the fray. I have seen just as many people who are an embarrassment to your side as to ours on various forums. If what we say on the net is to mean anything at all, we have got to encourage folks not to go that route.

The rest of your post was essentially off topic for this thread.