I am looking for lists (or repositories) of small molecules with connection tables (or machine-parsable molecular structures) which are Open. By Open I mean that anyone can, in principle download, copy or clone part or all of the site, re-use the information and redistribute without reference to the original site. At present I am aware of:

• Pubchem (10 million+ , superset of many Open datasets including NCI. I use this term to subsume everything at nih.gov)
• ChEBI (> 25 000 terms collected at EBI, not all with connection tables)
• MSD (ligands in Protein structures, collected at EBI > 5000)
• WWMM (250, 000 calculated structures from NCI database). Reposited in DSpace,
• Crystallographic Open Database crystal structuires collected from the literature or donated. Soon to be complemented with CrystalEye. This should give nearly 100,000 crystal structures.
• The BlueObelisk Data Repository (BODR). A collection of critical information collected by BO volunteers primarily as reference data for (Open) software. (includes non-molecular stuff like elemental properties). BODR is widely distributed on Gnome and other Open Source distros.

I’ve almost certainly missed some so please let us know. There should be a substantial amount of (molecules x attributes) in the collection and ideally it should provide complementary information to the above list. The connection tables (InChI, CML, SMILES, MOL) (or 3D coordinates) of the molecules should be easily accessible (i.e. not requiring resolving identifiers). Note that there are many useful sites which offer free queries on databases (e.g. NIST Webbook) but these do not fit the criterion of Openness – could they be downloaded in toto, cloned and redistributed (with attribution, of course). If we don’t already have it, perhaps there should be a page on the Blue Obelisk Wiki.

I am just relaxing in a hotel in Redmond, WA, US after two week’s very hard work instead of going downtown Seattle and shopping (which I hate unless it is for Obelisks). So the posts are going in all directions today. Incidentally a post seems to take about at least an hour which is why I don’t post every day. I am seriously considering audio blogs (podcasts) and would welcome advice. Part of my environment now is “digital libraries” – at least I get to interact with librarians, informations scientists, etc. There is a real buzz of change in the air. One feature of that is the Institutional Repository (IR) – a digital place where people in an institution (usually a University) put digital things.   That’s as close as I can get to the communality of vision. Why are people doing this – why are they NOT doing this? What do they want to put in? I’ll probably blog about this later. Caveat Lector is an intriguing blog from a librarian (Dorothea) an ex-publisher who describes herself as

Sure, I’m a geek. But I’m not a gadget geek or a Lib2 geek or even a web geek (more than incidentally). I’m not even really a markup geek any more; I sling XML now and then, but as a side requirement of my real work rather than the focus of my professional attention. What I am is a problem-solving geek. I have a problem with a technology (say, hm, I dunno, DSpace?), I beat the living daylights out of it with the nearest handy rock until either it does what I want or I decide that the problem needs a better tool than a rock and give up (complaining bitterly afterwards, of course).

She sounds like the sort of person that we desperately need – an XML-slinger who works in libraries and information science. We collaborate with Jeremy Frey (Southampton – CombeChem eCrystals, etc.) and yesterday that we agreed at the top of our shopping list we needed informations scientists embedded in chemistry (and other scientific) departments. That’s the role of the modern “library”. Anyway there has been a lot of debate about repositories and how to get stuff into them. As a scientist I know that this requires two conditions

• There has to be an overwhelming motivation for the scientist (like losing their job if they don’t do it) and…
• … it has to be trivially easy

At present neither of these are true. Recently JISC has announced a project SWORD to help people (authors?, librarians?) reposit material. That is a great advance. Jim Downing who advises me on much of this thinks it’s important. And I am supportive of what JISC is doing with repositories (readers should allow for the fact that they are supporting us!). So let Caveat Lector tell it her way…

Well, hot damn

I ask for middleware, and lo, there is middleware!

(No, I don’t think there’s a direct cause-and-effect relationship there. Even on my worst days I’m not that arrogant! Just shows that I’m not the only person with that particular train of thought.) Bring on the SWORD, y’all. I’ll wield that baby, you betcha.

Dies Martis, 24 Aprili 2007

Repository middleware

I did a lot of IR marketing this week, despite my perfect awareness that IR marketing doesn’t work. For a tactic that doesn’t work, I did manage to come away with some contacts, and it appears that the IR made its way into some heads, and that’s all good.

But if marketing doesn’t work, what does? Here’s the problem I’ve got: there’s a ton of material that’s IR-ready floating around, but I can’t get at it. My nose is mashed up to the window of other people’s hard drives, web servers, workflow silos, and collaboration tools. I want the stuff that comes out of those arenas. I just have no way to grab it. Here’s the problem everybody else has got: they need the curation, preservation, and “put this important content somewhere safe (but otherwise out of my hair)” tools that an IR theoretically provides, but they don’t need the hassle of extra deposit steps. They need an “Archive It!” button. They just have no way to build one… if they even know about the IR to begin with. I need middleware, and I need it badly. I don’t think DSpace or EPrints developers should be directly considering building the kinds of tools that Peter Murray-Rust is talking about. We’re the wrong people for the job (we can’t even do versioning!), and the job is being done elsewhere by others anyway, because faculty want and need these tools, and IT is finally listening. (I have direct evidence of that from my own job, but I need to keep fairly quiet about it because work is ongoing. You’ll just have to trust me.) What DSpace and EPrints developers should be considering is how to hook IRs up to the firehose of research products those other tools are producing. By my one-horse back-of-the-napkin calculations, that means an ingest API (no, not a command-line batch import tool, an API!) that is configurable enough to authorize certain tools for unmediated deposit and then prepopulate metadata fields with what those tools “know” about their content and the people who use them. It’s a tall order, but I dearly hope it’s not impossible, because I want to get my IR’s ingest pipe connected to that firehose.

One caveat (small C). If you build it they may not come.  That’s the challenge. If they do come, and it’s a good SWORD, they’ll be less likely to go away. That’s why we have to work on all areas of encouraging people to capture and reposit digital artifacts.

Jean Clause Bradley posts in Nature Island Review

I have blogged about the broken state of chemical information and the lack of semantics in current commercial chemical software and drawing tools. This is exemplified by the amount of incorrect structures received by publishers (and I have talked to several). What is the most common chemical substance received by a chemical publisher (or at least the Royal Society of Chemistry, thanks to Colin Batchelor)? It’s not water, or a common solvent. It’s an error. If you KNOW – i.e. I or Colin or someone else have told you – please don’t answer. But otherwise please guess.

I ‘ve pent some of the last two days talking with Steve Bryant who runs The PubChem Project. People often think (and I’ve been guilty of this) that there is a lot of junk in Pubchem (although the proportion is very low). That’s not really accurate – it would be better to say that there are a lot of name-to-structure links that most people would regard as inaccurate. The primary problem is names (and naming is one of the critical challenges of the digital age). Lewis Carroll recognised the central roles of names – in Alice’s Adventures in Wonderland Alice’s neck had grown extremely long and …

… a large pigeon had flown into her face, and was beating her violently with its wings.

`Serpent!’ screamed the Pigeon. `I’m not a serpent!’ said Alice indignantly. `Let me alone!’ [...] `And just as I’d taken the highest tree in the wood,’ continued the Pigeon, raising its voice to a shriek, `and just as I was thinking I should be free of them at last, they must needs come wriggling down from the sky! Ugh, Serpent!’ `But I’m not a serpent, I tell you!’ said Alice. `I’m a–I’m a–’ `Well! what are you?’ said the Pigeon. `I can see you’re trying to invent something!’ `I–I’m a little girl,’ said Alice, rather doubtfully, as she remembered the number of changes she had gone through that day. `A likely story indeed!’ said the Pigeon in a tone of the deepest contempt. `I’ve seen a good many little girls in my time, but never one with such a neck as that! No, no! You’re a serpent; and there’s no use denying it. I suppose you’ll be telling me next that you never tasted an egg!’ `I have tasted eggs, certainly,’ said Alice, who was a very truthful child; `but little girls eat eggs quite as much as serpents do, you know.’ `I don’t believe it,’ said the Pigeon; `but if they do, why then they’re a kind of serpent, that’s all I can say.’

This exemplifies a fundamental problem of naming – the pigeon uses phenotypes and Alice uses genotypes, and Alice’s phenotype is inconsistent with her genotype. I’ll try to create an analogy and then map it onto Pubchem. Mr Python sells musical animals such as parrots and mouse organs. He has a number of suppliers who send animals (and occasionally collections of animals) which are labelled. Mr Python is not an ornithologist, but he has bought a molecular biology kit and can sequence the DNA of the things he is sent. He uses the names the suppliers send and his own internal numbering system based on the DNA of the thing he is sent (let’s assume no intra-species variation in the DNA) running from C1…. He also has a cataloguing system for everything S1… Supplier 1 sends a live specimen labelled “Norwegian blue parrot” – its DNA is labelled C1 and its catalog is S1. He now gets:

• A “white mouse” C2 S2
• An “african grey parrot” C3 S3
• A box of assorted animals labelled “animal organ”. Mr Python cannot extract DNA and the label is * S4
• Another parrot labelled “norwegian blue” and with DNA consistent with C1. He labels this C1 S5

So far there is no problem. But now he gets:

• a bird called “oslo beauty” with DNA C1. He labels this as C1 S6.

Now when anyone asks for a “norwegian blue” or an “oslo beauty” it will retrieve catalog entries with DNA of C1. Mr python starts to regard these names as synonyms. When asking for this, most customers asks for “norwegian blue” and this becomes the preferred name. But now the confusion starts:

• he gets a picture of a parrot. Since he is not an art gallery he does not accept this into the collection.
• he gets a parrot labelled “norwegian blue” which does not look very perky. He puts this into his collection as S7. Try as he can he can’t get any DNA out of this. It is, in fact, a stuffed parrot. So he complains to the supplier – the rest is history – but the entry still stands in the record – “norwegian blue” – S7. He does not offer this to his customers.
• he gets a parrot labelled “norwegian blue” whose DNA corresponds to C3. There is a name collision, but Mr Python is completely ignorant of parrot names and it goes in his collection as “norwegian blue” C3 S8.
• He gets another bird labelled “parrot” with DNA C3. This is labelled as “parrot” C3 S9

Mr Python has kept an accurate record of what he is sent. He is deliberately impartial to what name belongs to what DNA. He does care about selling dead birds and does not offer them in his catalog. But for all the rest he simply shows what names have been associated with what DNA. If someone asks for “norwegian blue” they get offered both the C1 S1 and C3 S8. A request for “parrot” gets C3 C9 Pubchem has a similar approach. Here the role of the DNA is replaced by the chemical connection table or “structure”. Steve and his colleagues check the any sample offered to see if it has a connection table (CT). If this CT is already in Pubchem, they use its number. If not they create a new entry for the new connection table. If there are name(s) associated with the sample, these names are associated with the CT, no matter how apparently “incorrect” they are. The information provided by the supplier goes in exactly as it is sent. This lead to name-structure links which appear absurd to most of us. But we should remember that there have been some very strange names in the past. You might expect that “salts of lemon” is citric acid or a citrate, but actually it refers to Potassium Hydrogen Oxalate (Pubchem entry). So Pubchem records this accurately. The real problems is with chemical suppliers. The level of accuracy and checking is often extremely poor. Nick Day did a study on one compound (staurosporine) – which has only one canonical structure (determined by X-ray crystallography) but for which 19 variants in the literature exist (some are simply “wrong”, others are “fuzzy” – omitting stereochemistry, etc.). Formally the only way to resolve a naming problem is to convene a committee of the great and the good in the domain, have them debate endlessly and finally come out with a recommendation. That’s what the International Union of Pure And Applied Chemistry does. It’s a highly respected body, but such processes are slow. It took many years to agree on the name of element 104 (even though none of it exists in nature) because of US-Soviet politics. Often the names are never used in practice. We all talk of “water” – H2O – but that is anglophone and the systematic international name is “oxane”. But how many of you use that! So Pubchem records every name+structure association it has. For the name “Methane” it finds CH4 as the first structure. That’s because the preferred structure for “methane” is CH4. Pubchem generates “preferred” names and structures by a popularity algorithm (details of which are ?deliberately? not obvious). It also finds 15 other structures, including things like isotopically labelled methane, and also some “wrong structures” which have “methane” in the name but are not methane, nor do they have “methane” as a name. So this is a feature of the search algorithm – name searching is a hard problem – and as Pubchem continues to develop software will reduce. (There is no “absolutely correct” when you are dealing with linguistics). Let’s look at the names for CID 297. The preferred name is methane – it’s at the top of the list. But we also find “carbon”. Methane is not carbon, so why is “carbon” a name for methane? It’s because chemists often omit the hydrogen atoms when drawing or writing chemical structures. So in some contexts “C” is interpreted as CH4 (remember carbon has a valency of 4) and O as H2O. The fundamental problem is that chemoinformatics and chemical information is fundamentally broken and the main purveyors do not care. Chemical software tools will emit structures which are semantic rubbish, suppliers of  chemical compounds mislabel their bottles, databases of chemical information are closed and compete against each other in a commercial market. The semantics are necessarily inconsistent. So Pubchem faithfully reflects the broken nature of chemical infomation. It cannot mend it – there are only ca. 20 people – and anyway the commercial chemical information world prefers to work with a broken system. But could social computing change it? Like Wikipedia has? I talked with Steve  about this. He said said social computing had been tried on sequences and it hadn’t worked. Comments tended to be “I published that first but it has got X’s citation on it”. I think chemistry is different. And I think we could do it almost effortlessly – rather like the Internet Movie Database.  Here every participant can vote for popularity or tomatos. A greasemonkey-like system could allow us to flag “unuseful names” or to vote for the preferred names and structures. And this doesn’t have to be done on PubChem – it could be a standoff site addressed through a greasemonkey. Martin Walker tells us there are 23000 pages in Wikipedia on chemistry. If we link those to Pubchem – and Steve is happy for this to happen – then Wikipedia becomes a communal standoff annotation tool for Pubchem. Of course there are > 10million compounds in Pubchem, but many of them are probably uncontroversial – they probably come from a single supplier and are rarely accessed. Wikipedia addresses the most important ones.

In Correction/Retraction Notice Noel O’Blog (Noel was in our Centre until recently) shows how the Blue Obelisk Greasemonkey can show a richer view of the chemical literature. The greasemonkey, developed by Noel and others and reported in Blue Obelisk mailing lists and (Using Javascript and Greasemonkey for Chemistry – Bowiki), is a clever little thing. It sits in your (Firefox) browser and reads every HTML page you load. Everytime it sees some chemistry (including journal names) it recognises it can add links to the page. So the page becomes enhanced, enriched. Noel has applied it to the case of the chemistry retractions in ACS publications. When I tried to follow up ChemBark’s story I had DOIs but no links. With the BO greasemonkey these DOIs get translated to links which I can follow. In this way Noel has constructed the complete set of linked correction/retractions – there appear to have been at least 6 papers which have been affected. I think you can see from this that the blogosphere pays a positive role in helping post-hoc peer-review in chemistry. Noeal highlights the fact that anyone reading the original paper does not know that a retraction has been made. Obviously this is not possible in paper journals, but I would have expected a publisher to put up a note  saying “this paper has been retracted/corrected”. I am now thoroughly confused as to what I am seeing at the end of a DOI.  The fundamental questions are:

• Is a DOI and identifier to a static piece of information (which is what I would expect – as it stands for Digital Object Identifier) or
• Is a DOI a controlled addressing system managed by a purchaser of DOIs. IOW can a purchaser put different versions of the same information under the same identifier

If it is the former, then we have some chance of preserving the scientific record. If the latter, the purchaser of the DOI can rewrite history whenever they choose. Of course that is true of much web information. For example I sometimes edit a post on this blog after the first publication. Mainly to correct typos or add clarification. But, unlike a publisher, I cannot change the record. When this post is published it gets aggregated by Planet BlueObelisk which effectively acts as a record. (It’s actually because certain typos and styles foul up the Planet that I edit the post later). The posts are also captured by Technorati – and are probably cached in many other places. And these caches have timestamps. Admittedly this leads to a potentially confused set of versions but it is possible to reconstruct history. Unless publishing is made Open, history will be mutable.

I have not read the chemical blogosphere for some time (other than Blue Obelisk) and have been catching up with some of this in the plane. This is ChemBark from January and some my specific comments may be out of date, but the general questions remain. Scholars of scientific accuracy/fraud may wish to pursue the posts (which are IMO a valuable addition to the formal review process). === ChemBark == Interesting…

Posted by Paul on January 11th, 2007

— function createCookie(name,value,days) { if (days) { var date = new Date(); date.setTime(date.getTime()+(days*24*60*60*1000)); var expires = “; expires=”+date.toGMTString(); } else var expires = “”; var ck = name+”=”+value+expires+”; path=/”; document.cookie = ck; } function readCookie(name) { var nameEQ = name + “=”; var ca = document.cookie.split(‘;’); for(var i=0;i<ca.length;i++) { var c = ca[i]; while (c.charAt(0)==' ') c = c.substring(1,c.length); if (c.indexOf(nameEQ) == 0) return c.substring(nameEQ.length,c.length); } return null; } function URLEncode(plaintext) { // The Javascript escape and unescape functions do not correspond // with what browsers actually do… var SAFECHARS = "0123456789abcdefghijklmnopqrstuvwxyz-_.!~*'()"; var HEX = "0123456789ABCDEF"; var encoded = ""; for (var i = 0; i 255) { alert( “Unicode Character ‘” + ch + “‘ cannot be encoded using standard URL encoding.\n” + “(URL encoding only supports 8-bit characters.)\n” + “A space (+) will be substituted.” ); encoded += “+”; } else { encoded += “%”; encoded += HEX.charAt((charCode >> 4) & 0xF); encoded += HEX.charAt(charCode & 0xF); } } } // for return encoded; }; function getWLHost() { var hostdev=”pubsdev.acs.org”; var hosttest=”pubstest.acs.org”; var hostprod=”pubs.acs.org”; var localhost=window.location.toString(); if (localhost!=null) localhost=localhost.toUpperCase(); if (localhost.indexOf(“DEV.”)!=-1) { return hostdev; } else if (localhost.indexOf(“TEST.”)!=-1) { return hosttest; } else { return hostprod; } } function openURLRedirect(query) { var myframe=window.document.getElementById? window.document.getElementById(‘openURLFrame’): window.document.frames['openURLFrame']; var myform=myframe.contentDocument? myframe.contentDocument.openURLForm: window.document.frames['openURLFrame'].openURLForm; //var subscriberId=myform.subscriberId.value; var subscriberId=readCookie(“OpenURL_Saved_SubscriberId”); var url=”http://”+getWLHost()+”/wls/ACSOpenURLWeb/servlet/OpenURLRedirectorServlet?subscriberId=”+subscriberId+”&”+query; // document.location=url; // new window window.open(url,”openurl”,”toolbar=yes,menubar=yes,location=yes,scrollbars=yes,resizable=yes”); } var history_len1; var history_len2; function openURLSubmit() { // works with IE // window.document.frames['openURLFrame'].openURLForm.submit(); // this works with both IE and netscape history_len1=history.length; var myframe=window.document.getElementById? window.document.getElementById(‘openURLFrame’): window.document.frames['openURLFrame']; if (myframe==null) { window.setTimeout(‘openURLSubmit()’,100); return; } var myform=myframe.contentDocument? myframe.contentDocument.openURLForm: window.document.frames['openURLFrame'].openURLForm; if (myform==null) { window.setTimeout(‘openURLSubmit()’,100); return; } myform.action=”http://”+getWLHost()+”/wls/ACSOpenURLWeb/servlet/OpenURLServlet”; myform.submit(); openURLProcess(); } function openURLProcess() { var myframe=window.document.getElementById? window.document.getElementById(‘openURLFrame’): window.document.frames['openURLFrame']; var mydoc=myframe.contentDocument? myframe.contentDocument:window.document.frames['openURLFrame'].document; var myform=myframe.contentDocument? myframe.contentDocument.openURLForm: window.document.frames['openURLFrame'].openURLForm; var isOpenURL=myform.isOpenURL.value; var imageLocation=myform.imageLocation.value; if (isOpenURL==null || isOpenURL==” || isOpenURL==’unknown’ || imageLocation==null || imageLocation==” || imageLocation==’unknown’) { window.setTimeout(‘openURLProcess()’,100); return; } if (isOpenURL==’true’) { var i; for (i=0;i<999;i++) { if (window.document.images==null) break; if (window.document.images[i]==null) break; var image_name=window.document.images[i].id; if (image_name.indexOf('openURL_img')!=-1) { window.document.images[i].style.visibility=""; window.document.images[i].src=imageLocation; } } } // 12/14/2005 // go back to previous page so users don't have to click twice history_len2=history.length; var subscriberId=myform.subscriberId.value; createCookie("OpenURL_Saved_SubscriberId",subscriberId); var agt=navigator.userAgent.toLowerCase(); var version=parseInt(navigator.appVersion); if (history_len2-history_len1==1 || history_len1-history_len2==1) { // 4/17/2005 safari fix if (agt.indexOf('safari') == -1 ) { history.back(); } } }

–var SA_ID=”acspix;acspix”; (I have quoted this without permission and argue fair use). The publisher states that figures in the supporting information have been replaced. There is some doubt in the blogosphere as to whether an amended copy of the supporting information was added to the scientific record or whether the record itself was changed. If it is the latter then it is very serious. The ACS policy is: