A belated welcome to Sam Adams – another member of the UCC group. Although technically a graduate student “writing up”, that doesn’t do justice to Sam’s contribution both to the group and to the community. Sam combines a wide range of programming skills with a commitment to make contributions where they are needed. He will be well-know to people in the chemical Java world as the creator of JNInChI – a java wrapper for InChI. It’s probably fair to say that without this we’d have given up on integrating InChI.
Anyway, here’s a typical post…

Running and distributing Maven projects

Bill Hooker has expanded his views on funding from Microsoft and I’ll also comment on Peter Sefton’s comment…

Peters Murray-Rust and Sefton on “science and selfishness”

Peter Murray-Rust (welcome back to blogging!) has replied to Glyn Moody’s post about semantic plugins being developed by Science Commons in collaboration with the Evil Empire, which I discussed in my last post. Peter MR takes the view, with which I concur, that it’s more important to get scientists using semantic markup than to take an ideological stand against Microsoft
[PMR …]
Another, to my mind even more important, point was raised by Peter Sefton in a comment on Peter MR’s entry:

I will have to talk about this at greater length but I think the issue is not working with Microsoft it’s working in an interoperable way. The plugins coming out of MS Research now might be made by well meaning people but unless they encode their results in something that can interop with other word processors (the main one is OOo Writer) then the effect is to prolong the monopoly. There is a not so subtle trick going on here – MS are opening up the word processing format with one hand while building addons like the Ontology stuff and the NLM work which depend on Word 2007 to work with the other hand. I have raised this with Jim Downing and I hope you can get a real interop on Chem4Word.

(Peter S, btw, blogs here and works on a little thing called The Integrated Content Enviroment (ICE), which looks to me like a good candidate for an ideal Electronic Lab Notebook…)
There’s a difference between the plugins being Open Source and the plugins being useful to the F/OSS community. If collaborators hold Microsoft to real interoperability, the “Evil Empire” concerns largely go away, because the project can simply fork to support any applications other than Word.

Thanks Bill and Peter. To amplify some of the points.

• Our Addins are Open Source. We spent a great deal of time in negotiating the contract to make sure that all parties agreed. The Addins can be used without Word2007 (though some of the functionality such as “Open in Word” may then be no-ops.)  In principle anyone with a .NET environment can use and modify the Addin.
• We are working hard to make sure they are interoperable (that’s why we are working with Peter Sefton.)  Complete interoperabilityis  probably impossible at present but we have achieved a lot. Joe Townsend in SPECTRa-T has shown that it is possible to extract large amounts of semantic information from chemistry *.doc without requiring the functionality of Word. (A major problem is the binary legacy application files that chemists use – and this is where the real rough edges come).
• We are striving to make the add-in and the files it creates useful to the F/OSS community. The problem there is not Microsoft but the antiquated views of the chemical community. This is epitomized in the phrase “our software doesn’t read/write CML because no customers have asked for it”. We expect the reaction of many to be “Oh, it only reads and writes CML so it’s no use. When you add SDFiles and SMILES  [PMR: well-known legacy destroyers of semantics and creators of ambiguity and corruption] then we might take a look at it.” But I am religious on that. If the community wants to go 10 years backwards I am not going to help them. If they really want it we have written all the converters and yes, they are Open Source. In this battle Microsoft is an ally.
• I used the phrase “rough edges” in referring to using Open Office – I was writing in the context of creating documents with embedded semantic chemistry.
• Are we helping to cement MS’s monopoly? Possibly, possibly not. We are arguing and creating semantic operability. Our XML can be used by ICE and other Open systems. It will be harder to create a DOCX/CML file from a third-party system that can be read by Word. I believe it’s possible but hairy. My belief is that by promoting tha value of interoperable semantics then monopolies will start to disappear…

… just as Open Access will destroy the publishing monopolies.

One of the problems of having deserted blogging for months is that there is a lot to catch up with. So here’s the third post today. (I vowed not to post even once a day but…).  Here Egon reviews Chemical Markup Language. I’ll quote a bit and then comment…

Editing and Validation of CML documents in Bioclipse

One advantage of using XML is that one can rely on good support in libraries for functionality. When parsing XML, one does not have to take care of the syntax, and focus on the data and its semantics. This comes at the expense of verbosity, though, but having the ability to express semantics explicitly is a huge benefit for flexibility.
So, when Peter and Henry put their first documents online about the Chemical Markup Language (CML), I was thrilled, even though is actually was still SGML when I encountered it. The work predates the XML recommendation. As I recently blogged, in ’99 I wrote patches for Jmol and JChemPaint to support CML, which were published as preprint in the Chemical Preprint Server in a paper in 2000 in the Internet Journal of Chemistry. Neither of the two has survived.
Anyway, the Chemistry Development Kit makes heavy use of CML, and Bioclipse supports it too. Now, Bioclipse is based on the Eclipse Rich Client Platform architecture, for which there exist quite a few XML tools in the Web Tools Platform (WTP). Among these, a validation, content assisting XML editor. This means, I get red markings when I make my XML document not-well-formed or invalid. Just a quick recap: well-formedness means that the XML document has a proper syntax: one root node, properly closed tags, quotes around attribute values, etc. Validness, however, means that the document is well-formed, but also hierarchically organized according to some specification.
Enter CML. CML is such a specification, first with DTDs, but after the introduction of XML Namespaces with XML Schema (see There can be only one (namespace)). The WTP can use this XML Schema for validation, and this is of great help learning the CML language. Pressing Ctrl-space in Bioclipse will now show you what allowed content can be added at the current character position. …

Yes, CML is ca. 15 years old and was the first XML langauge to be published. It’s mature in that most of the functionality has been designed, and tested, and deployed. It’s complicated because (a) chemistry is complicated and (b) information is complicated. So uptake is gradual.
I owe Egon a great debt. He’s always believed in CML and has done more than almost anyone to make it happen. He’s developed a huge amount of Open Source. That makes it believable as well as giving people ways into using it.
It’s not universally used, but this is because XML is rarely used in chemistry. Chemistry is a very conservative area and probably at leat 10 years behind bioscience. the decisions in bioinformatics are made by scientists – the decision in chemical informatics are made by software and information suppliers. And most of those only innovate when their customers ask for something new. And since the customers are also conservative, progress is slow.
A common belief is that CML is jsut another format. That’s a very constraining view. the true position is that is it is:

• A component of semantic documents (no other format can manage that
• A modelling language. It can represent the internal data model and this is what we are doing in Chem4Word with microsoft.
• An information transfer tool. XML is the first system that can transfer information with almost no semantic loss. By contrast the legacy formats of chemistry are vastly lossy except within limited domains.
• An innovation tool. XML is extensible and so you can add you own extensions without breaking the original.
• A computational tool. We have actually developed CML for grammars (e.g. for generating polymer structures or the representation of imperfect information – generic and ambiguous molecules).
• A consistent and enduring component of the emerging web. We are actively developing ontologies and RDF. This is straightforward with CML – it’s impossible with SMILES, SDFiles, etc.
• A transducer for computation (e.g. using Toby White’s FoX).

There’s been a gradual adoption of CML – and I probably don’t know much of what has happened. The Open Source community has led this and all the main tools will emit or read it. The following companies have sponsored me to develop CML applications:

• Microsoft
• IBM
• Accelrys
• Unilever
• MDL

and I know of 3 I can’t mention who use actively it in their products.

And their are an increasing number of publishers. Pride of place is the Royal Society of Chemistry, but anyone who want to publish chemistry in semantic documents will probably use CML as there is no alternative. We know of one text book publisher who is using it for high-school chemistry.

And we always like to hear from anyone who has potential applications…

Glyn Moody and I share many of the same views but here is an area where we differ. His position is straightforward and logically defensible – mine is less clear but – in my own mind – tenable. Currently I am funded by Microsoft – is this compatible with being an activist for Open Knowledge? I reproduce Glyn’s post in full (see also one by Bill Hooker which shares some, but not all views.)

Open Science, Closed Source

One of the things that disappoints me is the lack of understanding of what’s at stake with open source among some of the other open communities. For example, some in the world of open science seem to think it’s OK to work with Microsoft, provided it furthers their own specific agenda [PMR emphasis]. Here’s a case in point:

John Wilbanks, VP of Science for Creative Commons, gave O’Reilly Media an exclusive sneak preview of a joint announcement that they will be making with Microsoft later today at the O’Reilly Emerging Technology Conference.
According to John, who talked to us shortly after getting off a plane from Brazil, Microsoft will be releasing, under an open source license, Word plugins that will allow scientists to mark up their papers with scientific entities directly.
“The scientific culture is not one, traditionally, where you have hyperlinks,” Wilbanks told us. “You have citations. And you don’t want to do cross-references of hyperlinks between papers, you want to do links directly to the gene sequences in the database.”
Wilbanks says that Science Commons has been working for several years to build up a library of these scientific entities. “What Microsoft has done is to build plugins that work essentially the same way you’d use spell check, they can check for the words in their paper that have hyperlinks in our open knowledge base, and then mark them up.”

That might sound fine – after all, the plugins are open source, right? But no. Here’s the problem:

Wilbanks said that Word is, in his experience, the dominant publishing system used in the life sciences, although tools like LaTex are popular in disciplines such as chemistry or physics. And even then, he says it’s probably the place that most people prepare drafts. “almost everything I see when I have to peer review is in a .doc format.”

In other words, he doesn’t see any problem with perpetuating Microsoft’s stranglehold on word processing. But it has consistently abused that monopoly by using its proprietary data formats to lock out commercial rivals or free alternatives, and push through pseudo-standards like OOXML that aren’t truly open, and which have essentially destroyed ISO as a legitimate forum for open standards.
Working with Microsoft on open source plugins might seem innocent enough, but it’s really just entrenching Microsoft’s power yet further in the scientific community, weakening openness in general – which means, ultimately, undermining all the other excellent work of the Science Commons.
It would have been far better to work with OpenOffice.org to produce similar plugins, making the free office suite even more attractive, and thus giving scientists yet another reason to go truly open, with all the attendant benefits, rather than making do with a hobbled, faux-openness, as here.

PMR: First the facts. My group are funded by Microsoft to work on Chem4Word – an Add-in for chemistry in Word2007 – and also as part of the OREChem communal project organizing semantic chemistry under ORE. I personally get no financial reward other than meetings in Seattle. The Add-in will be Open Source.

To state my position on the various potential concerns…

• Microsoft is “evil”. I can understand this view – especially during the Hallowee’n document era. There are many “evil” companies – they can be found in publishing (?PRISM), pharmaceuticals (where I used to work) Constant Gardener) , petrotechnical, scientific software, etc. Large companies often/always? adopt questionable practices. [I differentiate complete commercial sectors – such as tobacco, defence and betting where I would have moral issues] . The difficulty here is that there is no clear line between an evil company and an acceptable one
• Monopolies are unacceptable. I have some sympathy with this view. It’s not quite the same as above, as the categorisation is simpler. By this measure all monopolies should be opposed, regardless of whether they are evil or not – this includes Google – for example.
• Capitalism is evil and should be opposed. This is logically defensible but I don’t hold this view myself.
• Microsoft has abused the standards process. I’m prepared to believe this, just as I am prepared to believe that publishers hired Dezenhall to rubbish Open Access.

So why am I working with funding from Microsoft? The project(s) are aimed at developing semantic documents – just as John Wilbanks is doing. Microsoft Research believes in semantic documents for research and it is encouraging to find a supporter when the chemistry sector is still in the dark ages. At the end of this we shall have made significant progress towards linked, semantic science.

Is this perpetuating the monopoly? The monopoly exists and nowhere more than in in/organic chemistry where nearly all chemists use Word. We have taken the view that we will work with what scientists actually use, not what we would like them to use. The only current alternative is to avoid working in this field – chemists will not use Open Office.

For the record we also work with Open Office – primarily through Peter Sefton. That at least is encouraging alternatives. But – whether you like it or – Open Office has too many rough edges to make it easy for those who are not early adopters.

However monopolies do not last forever. I’ve lived hrough “Nobody was sacked for buying IBM”, “… DEC”. Substitute  “… Google,  Microsoft, etc.” I’m also optimistic that the hegemony of the commercial publishers will also crash.  Has the pressure of anti-trust, etc. has caused re-orientation in Microsoft? Are projects such as ours helping to promote change? Or simply adding legitimacy?

What I can say is that the individual people I work with in Microsoft Research are genuinely looking to engage with the scholarly community and to help promote Openness. What I have little insight into is the soul of the large corporate itself. I’ve worked for Glaxo, and am sponsored by Unilever and Microsoft (as well as having been funded by IBM). One can never be easy with large companies, but at the moment I can draw the lines.

From Sean McGrath’s blog:

Some time ago, I had some ideas burning a hole in my head. Ideas I knew I had no time to pursue but felt pretty passionate about. My Two Django ideas in need of good homes posting resulted in some meetings and some false starts…but it also resulted in meeting up with three ace Django/Python programmers / “recovering” scientists, based in the fine city of Cambridge, UK. Three of the smartest folks I’ve ever met to boot.
It took me all of, uh, 20 seconds to explain one of the ideas to them and the result, a mere matter of months later is http://timetric.com/. it is now in public beta and you might like to check it out.
The idea is simple but has significant consequences in my opinion
The basic thought is this: What if numbers were first class members of the Web ecosystem?…

• What if numeric quantities had their own “home” on the Web? I don’t mean a great big slab of numbers, or a database, I mean the actual numeric quantities themselves : the time it takes you to drive to work, the spot price of gold, the average daily rainfall in Tanna Tuva…whatever.
• What if the changes to those quantities are recorded through time? You would end up with a time series for each numeric quantity.
• What if each time series was a blog, with its own feeds, its own simple web based update mechanism etc?
• What if all these numeric blogs lived out in the cloud so that it can scale to silly numbers and provide very high availabilty?
• What if the entire system provided simple webby APIs so that developers can upload as well as download stuff easily?
• And last but definitely not least…what if new time series could be created using spreadsheet like formulae – and automatically updated when any of the underlying numeric quantities are updated.

That would be pretty interesting. Not a database on the web, not a spreadsheet on the Web. Something new and much more interesting on the Web.
It will be fascinating to see what types of applications get built on top of the Timetric platform.

The “three ace Django/Python programmers” are Toby White, Dan Wilson and Andrew Walkingshaw, with whom I worked for the last 3-4 years. Martin Dove, from Earth Sciences and Victor Milman from Accelrys conceived a project for automatically computing the properties of crystalline materials and roped in DTI, Frankfurt, IBM and Daresbury. The project (sadly) finished late last year.

Although we had tried to keep them on in various roles, last September the three aces told us that they were leaving to set up a company – it was very hush-hush and it was only a month or so ago that Andrew visited us and demoed what they were doing. It’s a novel idea and – as with all new web ideas – it’s impossible to predict how it will develop.

I resonate with “numbers having their home on the web”. The scientific semantic web will only come about when we can use numbers as easily as we use words. It’s remarkable how far words have taken us – we can use search engines to find words and phrases that describe many concepts but not numbers. We can’t ask Google (or Live Search) “find all countries with a population over 100 million”. And that’s only the first step. It’s important to have units of measurement “average temperature greater than 40” is not much use.

So Timetrics is a first step. It’s a specialised type of numeric info and it relies on people having created it in that form already. So I want it to be successful, both as a pointer to the future and also as something that I can feel I had a small part in. An important role of academia is to generate new forms of knowledge and – in many cases – those will turn out to be able to create wealth.

[And by the way, they also hack Fortran.]

As a memeber of UKPMC advisory, I’d like to highlight their blog. Here’s Peter Suber’s comments:
UK PubMed Central launched a blog on March 2.  Already it’s posted several developments which I haven’t seen elsewhere:

• March 2:  The Company of Biologists (COB) has adopted a hybrid OA option.  More, when COB publishes articles subject to funder policies requiring deposit in PMC or UKPMC, COB will make the deposits itself.
• March 4:  The European Journal of Human Genetics has adopted a hybrid OA option.
• March 4:  The British Library has been running workshops to alert life-science researchers to new developments planned for UKPMC between now and 2011.
• March 9:  The American Thoracic Society modified its OA policy to comply with the conditions required by the UKPMC Funders Group.

I’ve [PS] added the UKPMC blog to my daily crawl.

A very thoughtful and timely comment from Tobias Kind:

as you mentioned Chem4Word, what happens to such a (semantically enriched) word document as seen at http://research.microsoft.com/en-us/projects/chem4word/ when journals will convert it to a “flat” 2D bitmap “dumb” PDF. Basically information is lost. Most of the chemistry journals only accept PDF and Word but finally convert it to 2D PDF for DTP (desktop publishing) purposes. From the supplement sections of many chemistry journals which still provide structures or spectra or data tables as bitmap PDF (shudder/shiver) I don’t see any good coming (I agree that’s not a PDF problem and a simple ZIP file would solve the problem). Actually the process is more complex, as figures (containing structure and spectra) are processed independently from the text and tables are mostly included in the text.
…

One of the major shortcomings of the W3C effort is its failure to address compound documents and packaging. The web, with HTML and its embedding and linking, means that a resource is rarely a single entity but is composed of images, links to other documents, etc. We met this restriction when I submitted my first deposition to DSPace – it was an HTML document and I had to upload every image separately. Even then I couldn’t get the links to work.
So there are several de facto approaches:

• zip up the components
• download “web page” in proprietary form (such as mht)
• save document as *.doc (Word 2003 compound document)
• embed objects in PDF

These are relatively easy to do and most will accept a wide range of ASCII and binary components. The problems are:

• the contents are not semantic
• there are no semantics of the structure
• to reverse the process you need the same technology as used to create the package

In practice this means that reading is normally limited to recreating the same visual experience as the author had, often only on a limited range of platforms. It is uncommon to extract semantic components. Unpacking a foo.grot.bin will not enhance it beyond the original – often untransportable binary.
MIME types could have solved this problem but they aren’t up to supporting the complexities of the problem.
If you are happy to say “{corporation}  solves all my problems and I have a full range of authoring and reading tools and so do all my readers” where {corporation} ==Adobe or Microsoft, you can stop reading.
There is an urgent need to address the compound document problem, ranging from tight “single” documents with embedded components to “web sites” and even more distributed systems. We are involved in two such efforts:

• ORE. I’ll write another post on this… But it’s an Open semantic packaging system based on RDF. We are funded with others by Microsoft (OREChem)
• Chem4Word (disclaimer, we are funded by MS but I still speak my mind).  This uses the Microsoft Word2007 DOCX format (OOXML) which is a container for XML components (and proprietary binary). I am not going to defend DOCX as a wonder of clarity and Openness but it is an Open standard (whatever you feel about the process of arriving there) and – in principle – can interoperate with the Open Office “equivalent” ODT. In practice they don’t interoperate well but we are collaborating with Peter Sefton (blog) on authoring theses in ODT (funded by JISC). I think interoperability between ODT and OOXML will come slowly and undramatically. The fundamental problem is that its’ hard and neither ODT or OOXML has been well designed, certainly not for interoperability. Still we and Peter will persevere to developed semantic XML-based authoring systems. More later

===========================================================================
Tobias: PDF itself can hold XML or CML data, so PDF is not dumb by design, but nobody at the (put your preferred chemistry journal in here) seems to be aware of, or at least ignores that DOC or PDF can hold embedded XML data. Are you aware of any efforts from ADOBE or publishers to push semantics and chemistry into the PDF world? So how will Chem4Word stand against flat PDF or OpenOffice Write? What would happen if every reaction drawing, every spectrum would be required as embedded or attached raw/xml format? I guess there are too many different formats 🙂

Personally I am not a fan of using PDF as a packaging standard. The quality of PDF varies enormously and – yes – I have had correspondents on this blog who say all we have to do is buy Adobe tools and the problem is solved. I’m against single-vendor solutions in science, whether they be instruments, simulation programs, reagents or authoring tools. You get lockin which limits vision and innovation. You struggle with lazy vendors who don’t care. I haven’t tried to understand Adobe’s format.
But my biggest criticism of PDF is that it locks our thinking in the paper age. Why should documents have page numbers? why should be have two columns per page. Because the publishers force it on us. So PDF is a tool for constricting thought, not liberating it. Edward Tufte has similar views on The cognitive style of Powerpoint. He even argues (cogently) that the corruption of the message by PP was so bad that it was in considerable part responsible for the space shuttle disaster.
PDF corrupts and restricts thought. It encourages us to destroy semantics. Yes, it can – with great effort – hold semantic objects but nobody uses it to do so. That’s because PDF and PP make us lazy.
We are making progress Chem4Word so that it can author DOCX files containing chemistry. You may think I’ve just swapped one proprietary format for another. But the components are all XML standards (CML can hold molecules, reactions, spectra, spectral annotations, crystallography, compchem, synthetic recipes). All these are easily extracted from DOCX. So we can author with Chem4Word but use as XML. As with everything we do it’ll be Open (details later). But anyone can process DOCX files without Word2007 – it’s a zip containing XML components. It’s not fun, but Joe Townsend has already shown that we can extract a lot of chemistry from theses in normal *.doc files. Don’t take that as an excuse for laziness, because we have to move forward.
As for publishers there are small signs of change. We work very closely with the Royal Society of Chemistry and they understand the value of semantics. We’ll keep you in touch.

I am very proud to be part of the Open Knowledge Foundation – which has an amazing set of activities and a continually growing group of activists.

Founded in 2004 we’re a not-for-profit organization promoting open knowledge: that’s any kind of information – sonnets to statistics, genes to geodata – that can be freely used, reused, and redistributed. Find out more »
We organize events like OKCon, run projects like Open Shakespeare, and develop tools like CKAN and KnowledgeForge to help people create, find and share open material. See our full list of events » and projects ».

Much of this springs from the energy and application of Rufus Pollock who is a stone’s throw from the Chemistry Department. Rufus is not only energetic but (in the best possible way) goads others into action. So last Wed we met in the CB2 cybercafe in Cambridge where I managed to write a useful letter on behalf of OKF and also we continued to build up a website. I ‘ll reveal, but it’s great to watch Rufus hacking away at the Python and HTML/CSS. And there’s a range of other tools in his brain.
So here’s an advert for OKCON 2009 …

Open Knowledge Conference (OKCon) 2009

• where: Centre for Advanced Spatial Analysis (CASA), UCL, London, UK (map)

• when: Saturday 28th March 2009, 1030-1830

• programme: programme page

• register: register page

• cfp: call for proposals

• last year: okcon 2008

The Open Knowledge Conference (OKCon) is back for its fourth installment bringing together individuals and groups from across the open knowledge spectrum for a day of discussions workshops.

This year the event will feature dedicated sessions on open knowledge and development and on the semantic web and open data. Plus there’s the usual substantial allocation of ‘Open Space’ — sessions, workshops and discussions proposed either via the CFP or on the day.

Interested in giving a paper? Have a project to talk about? Want to run a workshop or session? Please see the call for proposals.

Want to get involved in putting the event together or otherwise helping out? Contact us at info [at] okfn [dot] org or add your name to the OKCon wiki page.

Last but not least: we encourage early registration as space is limited!

Great day out. Unfortunately I shall be abroad, but best of fortune…