In recent posts (e.g. Open Data – preservation) I have continued to raise the problem that “Open” should not just mean free of charge nbut also free to use. (Peter Suber calls these price and permission barriers).   So I am glad to see Jo Walsh making the point very strongly (Keeping “Open” Libre) and showing that much of the problem is because the English language cannot easily make the distinction. We are no in great danger that the same thing is happening to the word “Open”. Strict BBB language requires “Open Access” to remove permission barriers, but as Peter Suber says (“regrettably”) it is starting to become used for all sorts of lesser approaches which reduce the permissions. This is serious enough for Open Access where people spend huge amounts of energy and stress worrying about what they can and cannot do with published material. It’s even more of a problem for “Open Data” which is only just starting its career. We are still seeing very little evidence that people in the scholarly publishing community care about keeping data libre. Please prove me wrong anf I can include it in my article.

An interchange with a correspondent…

You [PMR] said in your Blog:

It is critical to distinguish between “Free” and Open. “Free”, in this context, simply means that the provider has mounted the data (not necessarily the whole data) on a web page. There is often no licence, no copyright, no guarantee of availability, no commitment to archival, no explicit freedom of re-use. The materials database is in this category – and to be fair it didn’t call itself Open.

The Open Knowledge Initiative says: 1. Access The work shall be available as a whole and at no more than a reasonable reproduction cost, preferably downloading via the Internet without charge. The work must also be available in a convenient and modifiable form. ================================ [Correspondent] This does not seem to go far enough in that if I have all good intentions and post material on a web server and then drop dead tomorrow the info will disappear pretty soon after that. Possibly lost forever! The distinction you make between “free” and “Open” suggests that Open means there is some permanency to the arrangement of having it available? Am I interpreting this correctly? How could this be monitored or managed?

PMR: You are absolutely right. I think the problem of preservation has not been addressed. Indeed until I starting thinking about it I hadn’t realised how relatively simple the preservation of text was and how difficult the preservation of data was. First: Who can one trust?It’s currently easy to deposit material with anyone – Google, Amazon, whoever. And to trust to spinning media to be replicated. But it’s very risky for long term preservation. There are many bodies working on this and the simple message is that it’s difficult, depends on what we want to preserve and how long we want to do it. There are many levels – the bitstream, the semantic content, the ontological context, etc. Places like the UK’s Digital Curation Centre understand and work on exactly this. [ Correspondent] is worried – like me – about the archival of chemical data iwithin the laboratory. What should be done? My personal answers are:

1. If the data is valuable enough an international data centre will store it. Biology is strong here and bioinformatics centres have an effective commitment to archival and also work on preservation. Chemistry relies on commercial and quasi-commercial organisations which generally accomplish far less.
2. My own inclination is towards global domain-specific repositories where the data are difficult and the volume merits it; national ones where the problem is understood but needs supporting (e.g. in mainstream chemistry) and where possible departmental ones (e.g. for crystallography, spectroscopy, computational chemistry.)
3. I am not a strong supporter that terabytes of data should be generally dumped in institutional repositories without good metadata and analysis software. Maybe future generations will welcome these hidden treasures and will have super-intelligent software.

I posted receently about the problems of describing Open Data -how strict should we be about boundaries? Peter Suber has replied What counts as open data?, Klaus Graf has also given an important emphasis on archiving in a comment to my post. Also Peter has blogged another example of an “Open Access” database: Jan Christian Bryne and eight co-authors, JASPAR, the open access database of transcription factor-binding profiles: new content and tools in the 2008 update, Nucleic Acids Research, November 2007.

Abstract:   JASPAR is a popular open-access database for matrix models describing DNA-binding preferences for transcription factors and other DNA patterns. [...] JASPAR is available [here].

In the last 2 months I have been thinking fairly furiously about Open Data and realising it can be considerably more complex than Open Access scholarly publishing. I’m certainly clear that the borderlines may have to be fuzzy, but not infinitely. Peter writes:

• Just a quick note on my offline talk with PMR about Material Properties, which I called “OA” in a blog post.  Neither of us could find its licensing terms, so we couldn’t tell just how open it was.  I needed (I still need, we all need) a generic term for such resources when we do know they are free of charge but don’t know any details about their licensing terms.  For better or worse “OA” has become that generic term, even while it has a narrower, earlier, more technical and more proper sense through the BBB definition.  I readily and often acknowledge that I use the term “OA” both ways –widely and narrowly, as a generic term and as the technical term for the BBB level of openness.  I also readily and often acknowledge that this ambiguity causes problems –see for example the Poynder interview at pp. 30-31.  I can add that I resisted this dual sense as long as I could and only acquiesced when it became an undeniable fact of actual usage.  For perspective, I’ve also argued that this kind of semantic spread is not a special calamity for our technical term, but affects most technical terms in wide use and needn’t prevent precise communication.
• One tempting solution is to come up with a new generic term so that “OA” can be limited to its strict BBB sense.  That’s desirable but difficult, since coining terms is not the same thing as assuring their use, let alone their intended use.  BTW, “free” would not make a better generic term, at least not yet, since it suggests to many people that a work is merely free of charge and does not also remove permission barriers.  A good generic term would cover all kinds of free online content, including those that are BBB OA.
• I share PMR’s hope that the term “open data” can stay fairly well tethered to its technical definition.  But the data world needs a generic term for the same reason that the publication world does.  If we had a good generic term for free online content, perhaps it could allow “open data” to remain univocal.

PMR: I agree with the sentiments here – but suspect we are both unclear of the way forward. I’m reluctant to use OA for databases since “OA” is already having to work pretty hard to manage the differences in practice and philosophy in scholarly publishing of articles (data is rarely included). “Open Source” has more or less got its act together, although there is both tension within the community of the Free/Open + viral schools; and also abuse of the term “Open Source”. I am keen to avoid the abuse of “Open Data” while it is still struggling to play a role. I n some disciplines “free” implies “Open”. In biosciences there is an unwritten agreement that freely available data is Open. Sequences, strucures, genes are made available usually without formal copyright or formal licensing.  There are thousands of databases with the same attitude as JASPAR (above) – our own MACiE database is similar. In all these it’s commonplace to download the whole data – for example we state “Each MACiE entry in the database can be downloaded separately as a CML file. This option is available from the left side panel, underneath the reaction step lists.” The bioscience community has a tradition of sharing and re-use which doesn’t need to be spelt out. Admittedly there is potential for confusion, and some databases do restrict their usage, And some will effectively have a no-commercial use clause. But there is a strong tradition of meetings where the principles are reinforced, where collaboration is made and traded. Generally it is expected that data will be re-used. In chemistry, in contrast, the tradition is  of gathering information and reselling it. There are no public Chemoinformatics Centres in the same way as Bioinformatics – in fact the Bioinfromatics Centres are steadily taking over the biological parts of chemistry. So by default it has to be assumed that any database on the web, however freely accessible at a point in time has no guaranteed permanency of access. There are often explicit barriers to re-use. So it’s important to have clear guidelines and clear labels – otherwise “Open Data” or “Open Access” is meaningless and acts only as a way of marketing warm feelings. This is more difficult because data are undervalued in the peer-esteem economy. A “paper” – however poorly read, however bad – even to retraction, is part of the sacrosanct “scholarly record”. Libraries and curation centres have a duty to capture this. In contrast there is nothing like the same obligation on any organisation to capture public datasets. Admittedly its’ harder, but that’s not the real reason. So I reiterate some guidelines. I’m still working these out and would welcome comment. (I don’t feel we should stray too far from the The Open Knowledge Foundation guidelines. ) As a start I would suggest the following:

• There must be some mechanism whereby the community could, if it wished, capture the resource for public archival without permission. This could be as simple as spidering the site, or a relational dump, or a massive file, or an iterator.
• There must be no permission barriers to re-use including commercial re-use.
• The data must either be the whole work (at a given point in time) or be clearly bounded (i.e. there should be no hidden data that the world cannot get access to in the same way).
• There should be no time limits on access and re-use.

Data is now acquiring the same power as software did two decades ago. It won’t be surprising if there are tensions – commercial, political, social. We need to identify and plan for them.

There are several reasons why I’m currently thinking about Open Data (see Open Data at WP for some collected wisdom and links). We’re currently collecting more chemistry data that we intend to make Openly available (see CrystalEyeknowledge base as an example). I’ve been asked to write an article for Serials review (Elsevier) on the subject and am putting my ideas in order. Chemspider announced Something New and Exciting Coming Soon… which contained an image with “Open Data” (no details). And Peter Suber announced New OA database on material properties, originally from the Chemistry Central blog which announced “The database is yet another of the free, on-line chemical services to have emerged in recent years. ” The use of “OA” was, I think, Peter’s. I didn’t agree with Peter in his description of Material Properties as an “Open Access” database, and I’m worried that we shall see the same imprecision in the use of “Open Data”. So I wrote to Peter and am amplifying the arguments here. As a baseline Peter and I are both on the advisory board of the The Open Knowledge Foundation (initiated by Rufus Pollock) which has developed the Open Knowledge Definition. I think it’s important to take this as a starting point for this analysis, thought there are aspects of databases which make the system much more complex. It’s good that the principle is simple to summarise:

In the simplest form the definition can be summed up in the statement that “A piece of knowledge is open if you are free to use, reuse, and redistribute it”. For details read the latest version of the full definition (with explanatory annotations).

I’m going to look at the most important clauses for science/chemistry (emphases are mine) – I have omitted other clauses but I adhere to them as well:

1. Access The work shall be available as a whole and at no more than a reasonable reproduction cost, preferably downloading via the Internet without charge. The work must also be available in a convenient and modifiable form.

 

3. Reuse

The license must allow for modifications and derivative works and must allow them to be distributed under the terms of the original work. The license may impose some form of attribution and integrity requirements: see principle 5 (Attribution) and principle 6 (Integrity) below.

 

4. Absence of Technological Restriction

The work must be provided in such a form that there are no technological obstacles to the performance of the above activities. This can be achieved by the provision of the work in an open data format, i.e. one whose specification is publicly and freely available …

 

5. Attribution

The license may require as a condition for redistribution and re-use the attribution of the contributors and creators to the work.

 

6. Integrity

The license may require as a condition for the work being distributed in modified form that the resulting work carry a different name or version number from the original work.

 

8. No Discrimination Against Fields of Endeavor

The license must not restrict anyone from making use of the work in a specific field of endeavor. For example, it may not restrict the work from being used in a business, or from being used for military research.

 

PMR: There are significantly different types of Open Data in science. There is raw data produced by the scientific experiment and increasingly published alongside “fulltext” or publications or theses. There is a curated, critical snapshot of a given experiment, perhaps images from a telescope or satellite. In this post I discuss the problems of “databases” or “knowledgebases” which are both fragmented and dynamic (e.g. CrystalEye and the Materials database.) It is critical to distinguish between “Free” and Open. “Free”, in this context, simply means that the provider has mounted the data (not necessarily the whole data) on a web page. There is often no licence, no copyright, no guarantee of availability, no commitment to archival, no explicit freedom of re-use. The materials database is in this category – and to be fair it didn’t call itself Open. A major problem, which we have discussed in some detail on this blog over CrystalEye, is that many databases are both hypermedia and dynamic. They are spread over many components and they change with time. Both CrystalEye and Materials fall into that category. It is technically difficult to make them easily available and there is no agreed mechanism for doing this. The work must be available as a whole. I agree this is critical, but it’s often difficult. Leaving aside the dynamic aspect there are a few possibilities.

• Bundle the data into a single “file” or a set of files. This has worked historically for the Protein Databank. The difficulties are that there is not usually a single simple object to bundle, and that it requires considerable maintenance.
• Provide an iterator over the data. This could either be a generic tool such as wget (which recurses over a hyperdocument) or a bespoke tool which is guaranteed to iterate over the data. This is the approach we have adopted (Jim Downing wrote it specifically to help the community download the data and has made it available under Open Source).
• Collaborate with a data provider (e.g. a Bioinformatics institute). This is a good approach if your community supports the idea of Open Data, but chemistry has yet to see the light

 

 

A few other comments. “convenient and modifiable form” and “no technological obstacles” cannot be defined precisely but I would ague that if the Open Data provider had published their formats and if there was Open Source code that would read the data that was sufficient. Note that for many files ASCII is sufficient if the metadata is well provided. There is no requirement for the Open Data provider to provide installation help for downloaders if the instructions are minimally clear. Open Access for scholarly publications implicitly guarantees certain aspects which are not guaranteed by default for Open Data:

• The whole of the work is available. This is almost always trivial for articles (but as we have seen is a problem for some sorts of data).
• There will be continued access to the work. This is based on (Gold) the permanence of Open Access publishers and the copying to inter/national repositories and (Green) the permanence of institutional repositories and in some cases inter/national repositories (self-archival on personal webpages does not guarantee permanent access). Repositories in general do not archive data.
• The work can be re-used. This is clear if a licence is embedded in the work or provided by the repository. Note that many repositories do not make the licence position clear.
• The work is in a convenient and modifiable form. Trivially readable for sighted humans. The rest is not always true.

Almost all these are major problems for Open Data. So I very much hope that we can use Open Data in a strict form which adheres to the Open Knowledge Foundation guidelines. This is a good time to cement or challenge them. But it would be a serious problem if we allow “Freely accessible” to become synonymous with “Open Data”.

Yesterday I needed the measured (i.e. not predicted) mass density for 2-bromo-propanoyl-bromide (CH3-CH(Br)C(=O)Br). This is a moderately common reagent and so I went to look for it on the Web – ultimately finding it on several sites. The value is ca. 2.061 g.cm-3 (many sites omit the units – argh!!). The temperature should also be reported – but isn’t. I need the measured density because many chemical recipes give the volume of reagents and I want to work out the molar ratios in reactions for which I need the density. I may also be interested in other measured properties such as boiling point. The problem is that it’s difficult to scrape these sites. They give little indication of copyright, are arcanely structured and often have poor semantics (e.g. units). The best known is the NIST Webbook, part of which reads:

• Thermophysical property data for 74 fluids:
  • Density, specific volume
  • Heat capacity at constant pressure (C_p)
  • Heat capacity at constant volume (C_v)
  • Enthalpy
  • Internal energy
  • Entropy
  • Viscosity
  • Thermal conductivity
  • Joule-Thomson coefficient
  • Surface tension (saturation curve only)
  • Sound speed

You can search for data on specific compounds in the Chemistry WebBook based on name, chemical formula, CAS registry number, molecular weight, chemical structure, or selected ion energetics and spectral properties.

---

NIST reserves the right to charge for access to this database in the future.The National Institute of Standards and Technology (NIST) uses its best efforts to deliver a high quality copy of the Database and to verify that the data contained therein have been selected on the basis of sound scientific judgment. However, NIST makes no warranties to that effect, and NIST shall not be liable for any damage that may result from errors or omissions in the Database.

---

© 1991, 1994, 1996, 1997, 1998, 1999, 2000, 2001, 2003, 2005 copyright by the U.S. Secretary of Commerce on behalf of the United States of America. All rights reserved.

It’s clear that this is not an Open site – most works of the US Government are required to make their works freely available but NIST has exemption for its databases so that it can raise money. Many suppliers list property information but scattered throughout somewhat uncoordinated pages. Moreover the copyright and crawling position is often not clear. My requirement is likely to be via robot – i.e. an asynchronous request for a property I don’t have, with the ability to re-use it without explicit permission. I am therefore wondering whether there are Open sites for chemical data that can be accessed without explicit permission. I am not interested in collections of millions of compounds, but rather ca. 10,000 of the most commonly used. A good source of data is MSDS (Materials Safety data Sheets), and here is part of a typical one hosted by  a group at Oxford University:

General

Synonyms: nitrilo-2,2′,2″-triethanol, tris(2-hydroxyethyl)amine, 2,2′,2″-trihydroxy-triethylamine, trolamine, TEA, tri(hydroxyethyl)amine, 2,2′,2″-nitrilotrisethanol, alkanolamine 244, daltogen, sterolamide, various further trade names Molecular formula: C₆H₁₅NO₃ CAS No: 102-71-6 EC No: 203-049-8

Physical data

Appearance: viscous colourless or light yellow liquid or white solid Melting point: 18 – 21 C Boiling point: 190 – 193 C at 5 mm Hg, ca. 335 C at 760 mm Hg (decomposes) Vapour density: 0.01 mm Hg at 20 C Vapour pressure: 5.14 Specific gravity: 1.124 Flash point: 185 C Explosion limits: 1.3 % – 8.5 % Autoignition temperature: 315 C

Stability

Stable. Incompatible with oxidizing agents and acids. Light and air sensitive.

It looks as if there are in the range of 5,000 to 100,000 compounds on the site – I haven’t counted and if so this is close to what I am looking for. It looks as if the creators are happy for people to download it – their concern is that it shouldn’t be seen as authoritative about safety (a perfectly reasonable request). If so, an Open Data sticker would be extremely useful and solve the problem. (There is the minor problem that there are no connection tables, but links to Pubchem should solve that). There has been talk of a Wikichemicals – and this is the sort of form it might take. It shouldn’t be too difficult to create it and the factual data on the pages doesn’t belong to anyone. So I’d like to know whether anyone has been doing this (measured, not predicted data) and whether there resource is Open.