Open science: competitions increase the quality of scientific prediction

I previous posts and comments we have been discussing the value of certain predictive methods for NMR chemical shifts. In the next post I am going to make a proposal for an objective process which I hope will help take us forward. Chemistry (chemoinformatics) is often not good at providing objective reports of predictive quality – the data, algorithms, statistics and analysis are often not formally redistributable and so cannot be easily checked.
In preparation for the suggestion here are some examples of how competitions enhance the quality of prediction:

CASP

Every 2 years (CASP1 (1994) | CASP2 (1996) | CASP3 (1998) | CASP4 (2000) | CASP5 (2002) | CASP6 (2004) | CASP7 (2006)) the Protein Structure Prediction Centre runs a competition:
“Our goal is to help advance the methods of identifying protein structure from sequence. The Center has been organized to provide the means of objective testing of these methods via the process of blind prediction. In addition to support of the CASP meetings our goal is to promote an evaluation of prediction methods on a continuing basis.”
There are independent CASP assessors who give their time on an impartial basis to oversee the procedure and judge the results of the predictions. Some more details:

For the experiment to succeed, it is essential that we obtain the help of the experimental community. As in previous CASPs, we will invite protein crystallographers and NMR spectroscopists to provide details of structures they expect to have made public before September 1, 2006. A target submission form will be available at this web site in mid-April. Prediction targets will be made available through this web site. All targets will be assigned an expiry date, and predictions must be received and accepted before that expiration date.”
As in previous CASPs, independent assessors will evaluate the predictions. Assessors will be provided with the results of numerical evaluation of the predictions, and will judge the results primarily on that basis. They will be asked to focus particularly on the effectiveness of different methods. Numerical evaluation criteria will as far as possible be similar to those used in previous CASPs, although the assessors may be permitted to introduce some additional ones.

There are four assessors, representing expertise in the template-based modeling, template-free modeling, high accuracy modeling and function prediction: In accordance with CASP policy, assessors are not directly involved in the organization of the experiment, nor can they take part in the experiment as predictors. Predictors must not contact assessors directly with queries, but rather these should be sent to the escramble(“casp”,”predictioncenter.org”)casp@predictioncenter.org email address.

and they follow up with a meeting.

Text Retrieval Conference

The TREC conference series has produced a series of test collections. Each of these collections consists of a set of documents, a set of topics (questions), and a corresponding set of relevance judgments (right answers). Different parts of the collections are available from different places as described on the data page (http://trec.nist.gov/data.html). In brief, the topics and relevance judgements are available at http://trec.nist.gov/data.html, and the documents are available from either the LDC (Tipster Disks 1–3) or NIST (TREC Disks 4–5), information on collections other than English can be found at http://trec.nist.gov/data.html.

A Third Blind Test of Crystal Structure Prediction

In May 2004 the CCDC hosted a meeting to discuss the results of the third blind test of Crystal Structure Prediction (CSP). The challenge of the competition was to predict the experimentally observed crystal structure of the 4 small organic molecules shown in figure 1 given information only on the molecular diagram, the crystallisation conditions and the fact that Z’ would be no greater than 2. The results of the competition are presented including an analysis of each participants extended list of candidate structures. A computer program COMPACK has been developed to identify crystal structure similarity. This program is used to identify at what positions the observed structures appear in the extended lists. Also, predicted structures obtained from the various participants are compared to determine whether the different approaches and methodologies attempted produce similar lists of structures. The hydrogen bond motifs predicted for molecule I are also analysed and an assessment made as to the most commonly predicted motifs and a comparison made to common motifs observed for similar molecules found in the Cambridge Structural Database

PMR: These havea range of objective (measured) and subjective (expert opinion) criteria for the “right” answer. The key components are:

  • the mechanism and evaluation must be independent of the competitors
  • all competitors must have an equal chance
  • the answers must be carefully created and hidden before the prediction
  • there is a closing date

It is essential that the data are Open and seen to be a reasonable challenge and that the analysis process is transparent. It is not essential that competitors software is Open.

This entry was posted in data. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *