petermr's blog

We are delighted at the practical and helpful  contributions from members of the community in helping to understand or correct outliers in the data set we are using. This is exactly what we hoped would happen at the start of the project and it has not started to gain momentum. I list some of them below to acknoledge the help. It is also highlighting the need for better tools for such collaborative projects – a blog is a poor mechanism but wikis also have their failings.
To reiterate:

• Nick has been through the dataset by hand and identified all data sets with potential misassignments or other anomalies. This has been done by comparing agreements within each set. A data set is likely to have been flagged if (a) it has a single widely outlying shift (b) two peaks (a, b) have coordinates yb, xa (as we have shown) giving an “X”-like pattern (c) has a large general scatter considerably greater than the average.
• Nick will post the major outliers based on RMSD. I don’t know how many there will be but I expect about 50 (hence the “20%”). These will be clickable – i.e. anyone with an SVG browser can imemdiately find our which peak is linked to which atom.
• After, and only after, these have been cleaned or accepted we will try to see if there are systematic effects in the data  – either the variance or the precision. We could expect that data from various sources could provide much of the variation, or the date, or the field strength, or the temperature, or the solvent. Unfortunately we do not have all the metadata as it isn’t present in the CMLSpect files.
• Finally we may be able to comment on Henry’s method. It is possible that certain functional groups have problems (Nick has some suspicions) but at present these are overwhelmed by variance from other sources in the experiment or its capture

So here are examples of useful comments. (I am not sure why Pachyclavulide-A is relevant – I can’t find it by name search in NMRShiftDB – but the effort is appreciated. However we are primarily looking for comments on the outliers we have identified.)

1. Egon Willighagen Says:
   October 26th, 2007 at 1:15 am eThe first one is another misassignment. Look up the structure in the NMRShiftDB and you will see one correctly assigned and one misassigned spectrums. This kind of issues should be filed as ‘data’ bug report at:
   http://sourceforge.net/tracker/?atid=560728&group_id=20485&func=browse
   I’m will do this one.
2. Egon Willighagen Says:
   October 26th, 2007 at 1:17 am eFiled as:
   http://sourceforge.net/tracker/index.php?func=detail&aid=1820353&group_id=20485&atid=560728
3. Wolfgang Robien Says:
   October 26th, 2007 at 8:48 am eAnother error: Pachyclavulide-A (should be C26 instead C27), MW=510
   Found automatically by the following procedure within CSEARCH:
   Search all unassigned methylgroups located at a ring junction. The methylgroup must be connected either with an up or down bond. As an additional condition, it can be specified if only “Q’s” are missing or if the multiplicity of missing lines can be ignored. I think a quite sophisticated check which goes into deep details of possible error sources. […]
4. hko Says:
   October 27th, 2007 at 12:02 pm eMisassignments NMRShiftDB (10008656-2) removed.
5. hko Says:
   October 27th, 2007 at 5:28 pm eMisassignments NMRShiftDB (10006416-2) removed. 45.0 and 34.4 reversed.

I posted our intermediate conclusions on Nick Day’s computational NMR project, and have received two lengthy comments. I try to answer all comments, though as Peter Suber says in his interview sometimes comments lead to discourses of indefinite length. I am taking the pragmatic view that I will mainly address comments (and subcomments) that:

• address our project as we defined it (not necessarily the project that others would like us to have done)
• add useful information (especially annotation of suspected problems)
• or show that our scientific method is flawed or could be strengthened
• relate to Open issues. In our present stage of robotic access to and re-use of data we can only realistically use databases that explicitly allow re-use of data and do not require special negotiation with the owners

There has been a great deal of discussion (far more than we had expected) on our project. Some of this has been directly relevant in responding to our direct requests for annotation of specific outliers and we acknowledge posts from Egon Willighagen, Christoph Steinbeck, Jean-Claude Bradley, Wolfgang Robien, and the University of Mainz. A lot of the discussion has been of general interest but not directly relevant to the aims of the project which was to show what fully automatic systems can do, not to create specific resources (“clean NMRShiftDB”). It is possible, though not necessary, that the work might be more generally valuable depending on what we found.

Wolfgang Robien: October 27th, 2007 at 12:03 pm e

You wrote: ….only Open collection of spectra is NMRShiftDB – open nmr database on the web.
Also the SDBS system can be downloaded – as far as I remember its limitited to 50 entries per day (should be no problem because QM-calculation are quite slow compared to HOSE/NN/Incr)

PMR: thank you for reminding us. I have not used SDBS and it looks a useful resource for checking individual structures but it is inappropriate for the current work as:

• robotic download is forbidden
• there is no obvious way of downloading sets of structures – they need to be the result of a search
• there is no obvious machine-readable connection table (there is a semantically void image).
• there are no 3D coordinates (this is not essential but it meant Nick could work almost immediately)

It is possible that if we wrote to the maintainers they would let us have a dataset, but this would double the size of the project at least.

If you need 500 entries with a certain specification (e.g. by elements, molwt, partial structure, etc.) and you want to perform a common project, please let me know …..

PMR: Thank you. This is a generous offer and we may wish to take it up. Contrary to your comment that I am an NMR expert, I’m really not – I’m an eChemist and this exercise in NMR is because I wish to liberate NMR from the pages of journals. If we find that Henry’s program needs more data or that yours has fewer problems it could be extremely valuable. We would wish the actual data to be Open so that others can re-use it.
PMR is quoted as “We downloaded the whole of NMRShiftDB. When we started we had NO idea of the quality. …”
That is quite true. There were a number of public comments on NMRShiftDB ranging from (mild) approval to (mild) disapproval, some scalar values for RMS against various prediction programs and some figures on misassignments, etc. These gave relatively little indication of the detailed data quality – e.g. the higher moments of variation.
If there is currently a full list of NMRShiftDB entries with your annotations this would be valuable. Currently I can find a number of comments on individual entries with gross problems at
http://nmrpredict.orc.univie.ac.at/csearchlite/hallofshame.html
but these seem anecdotal rather than a complete list.
PMR: and the second set of comments

ChemSpiderMan Says:October 27th, 2007 at 12:43 pm

Over the last 1-2 weeks Nick Day has been calculating NMR spectra and comparing the results with experiment. As there appears to be considerable interest we have agreed to make our conclusions Open on an almost daily basis. These lead to some compelling observations on the value of Open Data which I shall publish later. To summarise:
We needed an Open Data set of NMR spectra and associated molecular structures. The data had to be Open because we wished to publish all the results in detail without requiring to go back to the “owners”. The data also required annotation of the spectra with the molecular structure (“assignment of peaks”). Ideally the data should be in XML as this is the only way of avoiding potential semantic loss and corruption, but we would have managed with legacy formats.
There are well over 1 million spectra published each year in peer-reviewed journals. Almost NONE are published in semantic form – most are as textual PDFs or graphical PDFs. It is also unclear how many of these could be robotically downloaded without the publishers sending lawyers – at least Elsevier allow us to do this. In any case we would have to use OSCAR to extract the data, probably involving corruption and loss.
So we looked for Open collections of spectra. There are many, with an aggregated count of probably over a million spectra. However almost all are completely closed – they requires licence fees and forbid re-use. I have criticized this practice before and shall do so later, but here I note that the only Open collection of spectra is NMRShiftDB – open nmr database on the web. This has been created by Crhristoph Steinbeck and colleagues and contains somewhat over 20,000 spectra. Because Crhistoph is a member of the Blue Obelisk the data can be exported in CMLSpect (XML) without which Nick Day’s project would not have been possible.
We downloaded the whole of NMRShiftDB. When we started we had NO idea of the quality. This was deliberate, and part of the eScience- can a robot automatically determine the quality of data? Several correspondents have missed this point – It was more important to answer this question than whether the data were “good”. IF AND ONLY IF the data AND METADATA were of sufficient quality would it be possible to say something useful about the value of the theoretical calculations.
We knew in advance that certain calculations would be inappropriate. Large molecules (> 20 heavy atoms) would take too long. Molecules with floppy groups cannot be easily analysed. So we selected small molecules with rigid frameworks. This gave a starting set of about 500 candidates, each of which takes on average 1 day to calculate.
About 200 of these had computational problems – mainly that they were too large, they didn’t converge, Condor had problems managing the jobs, etc. So we have a final list of about 300 candidates.
We have listed the analysis of these results over the last few days. It is clear that some entries have “errors” and that there were also defects in the initial calculation models. Henry knew the latter, of course, but if we hadn’t known this at the start we would have been able to hypothethise that Br, Cl gave rise to serious additive deviation. So this is at least confirmation of a known effect for which we have made empirical corrections based on theory.
We have shown examples of poor agreements and anticipated all of them in principle. The data set contains many problems including:

• wrong structures. I am sure that at least one structure does not correspond to the spectrum
• misassignments. These are very common – probably 20% of entries have misassignments
• transcription errors. Difficult to say, but probably about 1-5%
• conformational problems. There are certain molecules which have conformational variability (i.e. floppy) but we have only calculaed one conformer. The most common example is medium-sized rings
• human editing of data leading to corruption. 2 entries at least

As a result Nick is going to produce a cleaned data set manually. He has already done most of this (has he slept?). He cannot do this automatically as the metadata are not present in the XML files. He will then be in a position to answer the questions:

• how much of the variance is due to experimental problems?
• if this is lower than, say, 70%, is it possible to detect systematic “errors” in the computational methodology.
• if so, can the method be improved.

If we can believe in the methodology then we can start to use it as a tool for analysing future data sets. But until then we can’t.
We realise that other groups have access to larger and, they claim, better data sets. But they are closed. I shall argue in a later post that closed approaches hold back the quality of scientific data.

Cameron Neylon has made a very useful comment on the Open Notebook philosophy which I can go along with:

Cameron Neylon Says:
October 26th, 2007 at 8:51 am eI’ve come in a bit late on this. I am with Jean-Claude and Bill Hooker I think. I would call this as it stands an ‘Open’ or ‘Public’ experiment rather than Open Notebook Science. This is not to say it is a bad thing. And the motivation for holding back a little on the data is a very good and reasonable one. There is also a grey area that Bill noted which is that obviously data is not made immediately available but that our approach is that it should be made available as rapidly as is practicable.

PMR: I also agree with this – so all protagonists including myself are agreed that what we are doing is not Open Notebook Science. I reiterate that it is our intention to make the protocols and data open as soon as practicable. Some of this is simply technical – we would like to write to wikis automatically but haven’t got the technology in place yet. By daily postings we are effectively making our evolving protocol Open.

I see the slogan ‘No insider information’ as a goal to work towards rather than necessarily achievable. It is a challenging one but it is what we aim for. We are working towards getting our analytical instruments to autopost to our blog so if I can make an analogy here. If a student of mine puts on an analysis overnight the results ideally would be published directly to the blog as they come off the instrument. It is possible that someone in Australia (or California) would see these, notice that we have discovered a new enzyme activity/new drg target inhibitor and then claim the observation.
We explicitly take this risk. In particular for some of the large facility experiments I am planning I will put up raw and partially processed data that it will take me some months to get through the analysis of – someone else may beat me to it. But if we think this through. They could claim the discovery (and to do so would have to do it rapidly – via a blog/wiki). They would have to refer to the dataset (because they won’t have the equivalent dataset) and so they would have to make the observation public in non-peer reviewed form. For the deliberate spoiler I think you can argue that there would be a rapid and very negative public response.

PMR: I agree that this is true for large public datasets that cannot be replicated. The particle physicists have a very carefully worked out protocol for when data can be released and who can work on it and who gets credit. So do some of the astronomers and geospatial communities. But chemistry has no tradition of releasing data (and much tradition of not releasing it) so we are encountering birth pains.

Two cases where there is potential difficulty. Someone being ‘helpful’ by making an observation that I would have made (basically the obvious conventional data analysis). This means you feel obliged to give credit. I would say this is still fine to include as a students work in a thesis but would feel obliged to give credit (authorship) in a publication. But there is clearly a very large grey area here. We want people to find things we’ve missed – this is part of the reason we are doing this. And there are many cases where someone sees something that is obvious in hindsight but it is very difficult to pin down whether you would have seen it unless you were looking.

PMR: Agreed. This is possible for CrystalEye – anyone is able to inspect our histograms of bond lengths and come up with stuff that we and others have missed. We really hope they do so

The second difficult area is when do you feel that data is ‘fair game’ for re-use. If I leave a piece of interesting data on the blog for six months and make no comment and publish no paper does this mean someone else can have a go and feel free to go with it, perhaps publish independently? 12 months? 18 months? I think there is a need to develop or evolve some sort of code of good practise here. We don’t want people having to ask permission every time before playing with our data – but we want them to play nicely giving due credit where appropriate. Perhaps we should tag datasets as ‘I’m done here – feel free to go at it’ or ‘Anybody got any ideas?’. I will try to post on this if I can find some time over the next few days.

PMR: These are useful suggestions. I would certainly intend that there was a “fair re-use” moment. ONS says that is at the moment of conception. The Protein Data Bank says it is at the moment of public release of the dataset which may be months after deposition – it takes time to go through the system and there are some embargoes (usually not more than 6 months).
Part of the problems of the current exercise – and why it isn’t immediately suitable for ONS – is that it would be possible for someone to replicate the whole work in a day and submit it for publication (on the same day) and ostensibly legitimately claim that they had done this independently. They might, of course use a slightly different data set, and slightly different tweaks. The other factor is that data in NMR seem to be so valuable – there are still daily comments on this blog from one group attacking another group (independently of us) – that it is difficult to be objective