Community peer-review? In chemistry???

Why do scientists publish in scientific journals? Are they still necessary? This has been debated intensely in recent years, but the chemical blogosphere gives a recent twist to the subject. Even if you aren’t a chemist you should be able to pick up the intensity and value.
The following seem seem to be the main arguments for having journals (I have added my prejudices). Please add your comments if you disagree. I’ll come to peer-review at the end.

• A formal record of the scientific process. Well, in part… But with the hamburger PDF culture much of the process is destroyed. Can the web do better? Look at Jean-Claude’s Useful Chemistry– for example a complete timeline.
• Establishment of priority. Yes, but… we could just as easily put the material in institutional repositories and I and colleagues have done for 250, 000 molecular structures. Our formal paper on this, which came out only about a year afterwards is not even Open Access so there is little point in referring to it here.
• Preservation of the scientific record. Probably true for most journals. But we have probably lost the Internet Journal of Chemistry (on which I was a board member) as it was closed access. Moreover the more we tend to “database of articles” rather than “journal” the less happy some of us feel.
• Widespread dissemination. Do people read more papers because they are in journals? Maybe
• A collection of articles with common or contrasting themes. This is still true in some places. But I suspect it’s decreasing with the mechanisation of science. How many physical chemists will now read a neighbouring article on synthetic organic chemistry? So why co-publish them?

So we now come to peer-review. There seem to be the following aspects:

• To detect (and possibly correct) errors in the manuscript.
• To decide whether the work is fit for some purpose. This could be relevant for clinical trials, etc.
• To give the work some element of esteem. This is normally a very crude measure – the paper is either accepted or rejected by the journal. (In some cases there is a “highlighted” paper – e.g. with a cover picture, etc.

Note that the peer-review system has nothing formally to do with the standing of the journal as in the Thomson-ISI “impact factor” (created and managed by a commercial organization). Nor with the citation count from the same source. All that peer-review can do is to aggregate certain articles in a given place and offer them to this “service”. If we believed in bibliometrics (which I don’t) then aggregation into artificial “journals” contributes little if anything.
Now the chemical blogosphere. I have commented on this earlier, but it is mainly created by graduate students and postdocs. Jean Claude, Christoph Steinbeck and myself are among the exceptions. So form your own opinions as to whether it is capable of any sort of useful peer-review.
Yes? Pasteur became a professor at 26; Kekule proposed tetravalent carbon at 28; Arrhenius proposed the electrolytic theory at 25… So I at least take the blogosphere seriously.
Let’s recall the conventional process of publication in synthetic organic chemistry. It’s worth remembering that this is a relatively self-contained field and (as I have commented earlier) concentrates on making molecules that are difficult rather than necessarily of wide general interest. (I will comment on this at a later date). The judgements that are made by the journal/publisher/reviewers are:

• have the authors made what they set out to make? and is their account believable?
• is the work novel?
• is the synthesis (not the final molecule) sufficiently interesting that it is worthy of publication?

Remember that tenure depends on whether the paper gets published in a journal of sufficiently high rating. The publisher makes additional stipulations (yes, the publisher – remember that they are running the process, not the scientific community).
• no part of the work can be reported before publication. This means that all Jean-Claude’s work is automatically barred from formal publication irrespective of its worth. I’m not clear what the reason for this is, but we have certainly been made aware of it ourselves by one publisher. It’s probably because they worry that they will sell fewer journals or no-one will read the papers.
• the manuscript is reviewed by (e.g.) three anonymous reviewers whose comments are passed to the authors who have the right to reply. This may iterate. The comments are normally not made public
• The editor (or editorial office) of the journal decides whether the manuscript is accepted. Authors may or may not try to argue with this.

The reviewers work hard, and for no financial reward. They do their best – it’s time-consuming (could take a day or more to review some papers). But they don’t always get it right…
TotallySynthetic in his blog has regular comments on synthesis papers (ca. 1 per day). There are between 10-20 useful comments on each from the blogosphere. Every month there is a vote on the best synthesis. So unlike the citation index, there is a considered judgement of worth by votes from the blogosphere. In terms of worth which would you choose? Blogosphere or Thomson-ISI?
Can the blogosphere spot things the reviewers missed? Here’s a recent TS post and blogosphere discussion of Stereochemical Reassignment of Mehta and Kundu’s Spiculoic Acid A Analogue:

Balwin, Kirkham and Lee. Org. Lett., 2006, ASAP. DOI: 10.1021/ol062361a.
Mehta and Kundu. Org. Lett., 2005, 7, 5569 – 5572. DOI: 10.1021/ol0521329.
Now this is a little scary. Both Sir Jack Baldwin’s group at Oxford and Mehta’s group at the Indian Institute of Science were working on the total synthesis of Spiculoic Acid A. Kundu published the Org. Lett. referenced above, last year, showing their progress towards the target, but it’s taken until now for someone to spot the problems with the paper. Baldwin’s group read the paper thoroughly as part of their publication preparation, and found quite a few problems.
Amazingly, Kundu assigned the stereochemistry quite incorrectly, originating from a Sharpless epoxidation where they predicted the wrong outcome… this in turn led to the incorrect configuration at many of the stereocenters. To compound this, stereocenters invert in configuration during the synthetic scheme for no apparent reason, and the nOe-NMR data looks distinctly suspect.
Come on ACS, lets referee papers properly! (PMR Note: Org Lett is an ACS publication)

Now let’s look at some of the comments on TS blog (I have excised the names for brevity)

The piss-poor refereeing has nothing to do with the ACS – they can only go so far. If group leaders and established, tenured faculty can’t be arsed to check stuff properly then things like this will happen.
What I find hardest to understand is how no-one noticed the benzyloxy stereocenter magically inverting (although it probably has something to do with the appalling quality of the transition state diagram)
[…]
How can you miss things as obvious as the outcome of a Sharpless epoxydation? How can you erroneously change the configuration of a benzyloxy center from one reaction to the next one?
Don’t get me wrong. Errare humanum est. But before publishing data, you should make sure that no gross mistakes remain. Other than that, it’s the referees’ fault.
Another question is, was this work thoroughly refereed at all? I mean, there is a 2-month time-span between reception and publication (from September to November!), so it should be enough time for correct, careful refereeing.
Last time I sent something to Org. Lett., one of the referees sent a three-page list (!!) with corrections and changes he thought should be made to the paper and supporting info. I suppose you don’t assign “picky” referees like that to every paper (specially for “renowned, established group leaders”), but I’m pretty sure that at least, that guy would have spotted all this mess way before it got to the point it did…
[…]
Part of the problem with manuscript refereeing is the turn around time – with OL you get a week. If it is a long synthesis, then it requires a significant amount of time to go through all of the data carefully. Some reviewers do not take the time to do this – if the paper looks ok, and the chemistry appears to meet the standard (whatever that may be), then it gets recommended for acceptance. I have reviewed several manuscripts from well-known practioners, and they do not get a free-pass from me!
[…]
The paper by Mehta had pointers that a curious and careful referee could have investigated.

http://sanjayat.wordpress.com/2006/11/01/self-correcting-science/
Unfortunate that the new structure could be obtained using the information given in the paper itself, with no additional experiments being conducted.

[…]

TotallyMedicinal. I wouldn’t say it has ‘nothing to do with the ACS refereeing’. The mistakes in that are so basic that and as they were missed it makes me wonder what the referees actually do with communications of total synthesis? I am no referee but I would imagine it is common sense to run through each individual step regarding their integrity etc?

[…]Speaking as someone who has refereed papers for Org. Lett. and other ACS journals, I’m not sure how much to blame the refs here. Of course, once you know there’s an error, it’s difficult to assess whether you, as a referee would have spotted it. I’d like to think that I’d have noted the mysterious inversion of stereochemistry in the Diels-Alder reaction, but I’m pretty certain that I wouldn’t have noticed the wrong enantioselectivity in the Sharpless… it’s just one of those things where you assume the authors knew what they were doing.
As for re-interpreting someone’s NOESY spectrum – well perhaps I’d have looked at it if I’d noticed the anomaly later on, but almost certainly not otherwise.
One of the problems for referees is that it takes a really long time to do it thoroughly… and for what reward? As one journal editor put it to me once – “It’s like peeing yourself in a pair of dark trousers – it gives you a warm feeling, but no-one else knows about it.”
[…]
This issue also raises the question about how referees can actually do a good job. The problem is of course that there is an exponentially rising amount of information out there waiting to be refereed, and most of it is uninteresting and ordinary. Plus, the referees have their own stuff to do. Is it really possible to have foolproof refereeing, even of simple things? There’s got to be some things that referees have to take on faith, even though ideally they should not do it.
[…]
Mike’s raised a very important point. Thorough refereeing is really an ungrateful job. It will take a lot of time and sometimes you’ll just spend 3 hours carefully checking a paper, Supporting Info and all NMR spectra included, to find out that there’s nothing wrong with it, other than a few mispelled words and a couple of problems in English.
2006 seems to be a fertile year in this kind of mistakes and fait-divers! We would be discussing the whole issue of refereeing all over again… any plausible alternatives?
[…]
Possible alternative: the model proposed by the Public Library of Science (http://www.plosone.org/) sounds exciting, coupling good old peer review system with open-access community-based reviewing/commenting/rating.

Curly Arrows runs another blog:

Now when I read this paper it actually came across as a nice piece of synthetic work. Unusually, these guys blatantly admit that their synthetic strategy towards the natural product failed. So what they do instead is provide a proof of concept by synthesising an analogue of the natural product using some Diels-Alder chemistry. Okay so that’s all fine and dandy. At this point I’d like to say that I am very glad that I didn’t referee this paper because things are about to get very hairy. Moving swiftly on to 2006 where Baldwin and co-workers publish the total synthesis of the enantiomer of Spiculonic Acid A in Chemical Communications
Biomimetic synthesis of marine sponge metabolite spiculoic acid A and establishment of the absolute configuration of the natural product
James E. D. Kirkham, Victor Lee and Jack E. Baldwin, Chem. Commun., 2006, p. 2863
–
A very nice piece of synthetic work and a well written paper too. These dudes at Oxford really know what they are doing. So at this point I guess that Baldwin and his mates realised that there were some discrepancies between their data and those of Mehta and Kundu. Hence, they decided to sit down and dissect Mehta and Kundu’s paper to figure out what was going on. The result of this little exercise was published in Organic Letters recently:
–
Stereochemical Reassignment of Mehta and Kundu’s Spiculoic Acid A Analogue
Kirkham J. E. D., Lee, V. and Baldwin, J. E., Org. Lett., 2006, ASAP Article
DOI:
–
Now this paper is really worth a read. We are talking major bitch slapping here. To me the most unbelievable mistake is the incorrect stereochemical assignment of an epoxide obtained by a Sharpless asymmetric epoxidation.
It appears that these Indian dudes haven’t been able to use the mnemonic model published by Sharpless to predict the stereochemical outcome. This is what Mehta and Kundu write in their paper regarding the epoxidation:
–
Sharpless epoxidation of allylic alcohol 19 in the presence of the D-tartaric acid diethyl ester was stereoselective (9:1) and afforded the epoxide 20 in a predictable manner with ample precedence.
And that’s only the beginning. Their NOE interpretations are all over the place and it seems that they can’t decide on the final stereochemistry of their Spiculonic Acid A when you compare the structures in the supplementary material with those given in the paper. Here’s another brilliant quote from Mehta and Kundu’s paper regarding their NOE interpretations (notice the language. One of these guys must have spent some time in the US and bought himself a dictionary):The stereostructure of 9 was delineated on the basis of incisive analyses of its spectral characteristics, particularly the COSY and nOe data.
Anyway, hats off to Baldwin and co-workers for spotting all the mistakes and submitting the paper and to Organic Letters for accepting it. It is quite remarkable to think that these guys from Oxford have managed to publish in Organic Letters without conducting a single experiment. I highly recommend reading these three papers in chronological order.

So – in my estimation – this is effective peer review. The community knows its stuff, read the literature. What would we lose if experiments were made public as Jean-Claude does? And why shouldn’t we use the community vote on the value of syntheses? But I suppose I am too romantic

: