In today’s Science magazine John Bohannon describes how he created a fatally flawed fake science paper and sent it to 304 on-line “open access” journals of lesser repute. Despite the claims that these journals were peer-reviewed, 157 accepted the paper and 98 rejected. The rest have not been heard from and may be among the missing.
Open access journals mostly provide free on line access to their contents, and recover their costs and make profit by charging publication fees than can be up to several thousand dollars. While the article implies that this is a practice limited to shady on line journals such as the Journal of Nature Pharmaceuticals, many top drawer print journals also charge page fees to reduce the enormous cost of publishing technical articles.
So what Bohannon did was create 300 slightly different articles (using a sort of Science Mad Libs approach to fill in the blanks) but all with the same critical flaws, such as a figure that showed exactly the opposite of what the article claimed regarding dose-response.
Spoofing the spoofer, Michael Eisen, the creator of the highly respected Public Library of Science (PLOS) journals claimed in a blog post to be the author of a fake article that reported on bacteria using arsenic where phosphorus normally occurs, and speculating on the extraterrestrial consequences. This is a joke on Science who actually published that article, which was later found to be incorrect according to a following study published in Nature.
Bohannon’s “sting” was called “flawed” by Martin Eve, writing in Medical Press, who points out that Bohannon singled out open access journals, when there were plenty of weak print journals who might have fallen for the scam as well. In fairness, Bohannon does note in the Coda to his article that he only eliminated print journals to make the task tractable, and he might have gotten the same result from questionable print journals.
Many of the journals Bohannon targeted were on Beall’s List, a list of questionable or predatory journals compiled by librarian Jeffrey Beall, and the rest from the Directory of Open Access Journals (DOAJ) which are supposed to be more reputable. Some were on both. A lot of the questionable journals, even though they purported to be American, were actually located in third world countries and were clearly in it for the money.
So what does this mean? Is a peer-reviewed science hopelessly compromised? Probably not, if you eliminate the obviously junk journals that spring up like mushrooms after the rain. One easy test is whether the journal has been indexed by PubMed, the journal abstracting service managed by the NIH. You would not find the specious Journal of Nature Pharmaceuticals there. (Although we understand it has now been shut down anyway).
Another thing you can do is look for the journal’s impact factor, which is a measure of the average number of citations made to articles in that journal by articles in other journals.
This, for example, would help you understand that the Journal of Organic Systems, where Judy Carman published her now discredited study on feeding pigs GMO crops was published. It is not listed in PubMed, has no impact factor, and is supported by the Organic Federation of Australia. Despite this fact, references to her paper still occurs on the Consumers Union Web Site, raising questions about their judgement on refrigerators and lawnmowers, too.
You might also be suspicious if you are looking at Volume 1 Number 1 of a journal, such as the poor paper on Hematoxoicity of Bt published in the first issues of the previously unknown Journal of Hematology and Thromboembolic Diseases.
If you’ve ever tried to publish in a peer-reviewed journal, or acted as a referee for one (I’ve done both) then you know how hard the journal editors and the authors work to make the papers the best they can possibly be. Sometimes there are 4 or 5 or more back and forth revision requests before the paper reaches the journal’s standards, and sometimes you just get rejected. The quality of articles in most journals is very high indeed despite these pranks on low-level journals.