Revamping the funding system (Ioannidis)

In this week’s Nature (not sure whether the scholarly poor have to pay for this so I précis a bit) there’s a useful review More time for research: Fund people not projects
John P. A. Ioannidis Nature 477, 529–531 (29 September 2011) doi:10.1038/477529a. It highlights the failures of the current system – huge amounts of effort expended by applicants and reviewers and there is great dissatisfaction. The current trajectory may make things consistently worse. So Ioannidis gives a table of possibilities. I reproduce bits of it here without permission (I am visiting NPG on Monday so they can throw me in their dungeon if they want):

Option ++ Pros –- Cons xx Example ?? Who would be funded

Egalitarian (fund everybody) ++ Avoids peer-review biases Gives sufficient amounts to scientists doing low-cost research Small administrative burden — Does not support large research efforts Does not recognize exceptional scientists xx Some universities fund the salaries of all their faculty?? All.

PMR: This is how I started (1967). It was called the dual-support system. You could rely on normal lab equipment, consumables, travel, and access to technical support. We were very well supported. It was actually so lavish that you were never brought to account – you could do what you wanted. I did a lot of stupid and pointless stuff. But I was also able to build the core of the chemical informatics program that I am still pursuing.

Aleatoric (fund at random) ++ Avoids peer-review biases Small administrative burden — Will not capture all deserving scientists xx Foundational Questions Institute ?? Flexible

PMR: For small amounts of funding with a large number of applicants this may be useful. For example if I have a 15% chance of getting a summer student funded I’d be happy to go into a lottery. But it doesn’t scale.

Assessment of career ++ Captures career trajectory Has gold-standard status — Is vulnerable to favouritism Inappropriate for young researchers Is labour-intensive xx MacArthur Fellows Program ?? Few elite scientists (or else administratively burdensome)

PMR: I was awarded a CIBA-GEIGY fellowship for a sabbatical year with Jack Dunitz when I had little formal track record (DPhil + 6 years in post). It changed my life. It wasn’t really assessment of career as much as assessment of promise. I hope I have fulfilled some of that.

Automated impact
indices ++ Eliminates favouritism Evaluates many applicants with ease Approaches objectivity — There are many indices, all with flaws; no consensus about best one to use Indices can be gamed Databases have shortcomings (such as imperfect citation coverage, entry errors, name disambiguation problems) xx UK Research Excellence Framework ?? Flexible

PMR: The only attraction of complete automation is efficient bureaucracy (and we know how often automation fails). It has the same objectivity as an income tax form. Formally correct but utterly depressing. It favours regression to the mean. There is no creativity in funding.

Scientific citizenship ++ May improve science, if good practices are rewarded and bad ones penalized — Automation is not yet possible for data gathering, and is difficult for some citizenship practices
Has peer-review biases xx Financial incentives to peer reviewers ?? Could be extended to many scientists only for aspects that can be automated

PMR: quoting without permission:

Funding systems could reward good scientific citizenship practices, such as data sharing⁴, high-quality methods, careful study design and meticulous reporting of scientific work⁵. Openness to collaboration, non-selective publication of ‘negative’ findings, balanced discussion of limitations in articles and high-quality contributions to peer-review, mentoring, blogging or database curation could also be encouraged. Researchers might be rewarded for publishing reproducible data, protocols and algorithms. However, some citizenship practices are difficult to capture in automated databases, so would be subject to the disadvantages of peer assessment.

I am not sure “Scientific citizenship” is the best term – this is as much about multivariate indicators of value and esteem (i.e. going beyond the mindless “how many papers and how often cited”). It can be gamed (probably easier than citations). However it’s obviously something that must be pursued and rapidly but overlaps with several other approaches.

Projects with broad goals ++ Proposals are easy to write and review Formulating work can be flexible Permits targeted innovation — Does not eliminate project proposals Is vulnerable to favouritism Holds potential for exaggerated promises and claims xx NIH Director’s Pioneer Awards Howard Hughes Medical Institute ?? Few elite scientists

PMR: I think there are two axes here – breadth / freedom, and patronage. I’ve had patronage and it can be extremely valuable (though often not meritocratic). Patronage often devolves to selected institutions, e.g. in the eScience program Cambridge was awarded a Centre. This allowed people in Cambridge to apply selectively for funds and I happened to be in a position to get 6 PDRA years fairly easily. Similarly there are often targeted programs designed around a group of institutions – if you happen to be in them it’s much easier to get funding. Perhaps the most specific was the Cambridge-MIT Institute (http://www.cmi.cam.ac.uk ) which “helped develop DSpace – a groundbreaking future-proof digital archive” (their words, not mine). I didn’t directly get funding but it helped to get JISC funding. And in eScience there were several projects where I could become a Co-I – eMinerals, materialsGrid.

But my biggest patron was Unilever and to them I am very grateful. They provide(d) infrastructure, project funding and a lot of freedom. Patronage is not egalitarian, can be abused, and can be misdirected. Similarly I regard Microsoft as a having elements of a patron. The long association through eScience makes it much easier to agree new mutually beneficial projects.

There are other models. I think cooperatives/collaborations (e.g. Mat Todd’s Open Drug Discovery and Open Source Drug Discovery in India) are valuable new models and can mix public and private funding. And while not everyone can receive money, there is greater opportunity for spreading the net and reaching out to citizen science. Similarly some of the funding that the OKF receives allows for considerable freedom and I applaud this – even more because this clearly supports a common good rather than fostering a single person’s career. It’s probably peripheral to mainstream research but it’s often still research.