More "potential reproducibility"

I’ve just blogged about a group (Open Data is critical for Reproducible Research) which is wanting to publish reproducible science. Quite by chance I then read  Bill Hooker reporting another effort to create reproducibility:
BillH: Openness is spreading, one researcher at a time: Jeremiah Faith, a Boston U graduate student in bioinformatics, has put his lab notes online:

Open Notebook Science […] is a term coined by Jean-Claude Bradley. The idea is simply that the heart of every person’s research – their lab notebook – should be open to the world.Since most of our scientific work is funded by tax payers who expect their money to be well-spent, it’s interesting that openness isn’t required. Science typically builds on the body of available knowledge – the more knowledge available the faster science goes. It’s striking when you visit other labs in person; you see all of their unpublished work, and you know that most of their results and data won’t be available to the bulk of the scientific community until a year after each particular scientific project is finished. By the time papers are in print, it’s old news to the insiders. More striking is when you visit labs whose work you’ve thought about replicating and expanding on. It’s not too uncommon to find that only one person in the entire lab is able to get the technique to work, and even for him the technique only works on Wednesdays. This type of information would be useful to know before you embark on a useless three months trying to adapt their method. But scientific publications are covered in a thick coat of high-gloss finish, making these unacknowledged difficulties hard to detect.
Lab notebooks on the other hand are flat black. As long as people keep them regularly updated, they contain the good, the bad, and the completely nonsensical results.
Today I test the waters of Open Notebook Science.
The latest version of my lab notebook is now automatically posted on J’s Lab Notebook Page each night. I’ve been using an electronic lab notebook for two years now, so there’s quite a bit of data in there – good and bad (300+ pages).

BillH: This is simply fantastic. One of the things that Open Science advocates most sorely lack is concrete examples. Doing research in public, instead of in secret, is a new and somewhat unnerving idea for most scientists; early adopters like Jeremiah are essential to take the edge off that unfamiliarity.(It’s also, to be honest, just plain fun to snoop around in someone else’s lab notes! I was amused to note that Jeremiah talks to and about himself in his notebook, the same way I do — “if I weren’t so stupid I’d…”, “next time load the control first, doofus”, etc. I wonder if everyone does that?)

PMR: This is simpy fantastic. An interesting thing is that this is zero-cost innovation. J-C coined the term “Open Notebook Science” after he had posted to the Open Data entry on Wikipedia and he and I had discussed possible terms. So after J-C blogged about it and set up his own site, Jeremiah used the idea to help think about, and promote, his own desire to communicate.
This is the power of the blogosphere – a good, well fashioned, idea can spread. Young scientists care about reproducibility. They are the ones that suffer from made-up, fuzzy, under-reported material. Do the lab heads and professors encourage reproducibility? I expect that most do but the pressures to publish in high-impact is too great to be ignored.
Jeremiah and J-C are gambling their scientific future on publishing their work openly. Some “high-impact” publishers would have knee-jerk reaction and refuse to consider manuscripts because the work had already been published. This would be rubbish. Any responsible science publisher should take very seriously the requirement to publish good science. Science that is created in an Open environment adds an extra dimension to “good”. It isn’t necessarily better, but it has less chance of being worse.
So, publishers, give us something to suggest that you are excited by these new developments. That you want to change science practice for the better. I’m guessing that Open Science (sensu J-C, not RSC) will help to promote the work better. And isn’t that what science publishing is about? Communication, not hiding.
And J-C and JF I can’t promise anything but I would bet that by gambling on Open Science you will be doing yourselves a lot of good. The world is waiting for this and will welcome you.

This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to More "potential reproducibility"

  1. Peter,
    As always, I appreciate your support.
    What has become very interesting is the dialogue sparked by the Open Notebook/Open Source Science debate. You say the main point about publishing science is communicating it. But then we have seen from Heather Piwowar how difficult it is to get raw data from many researchers.
    I wonder how many journal articles can survive the scrutiny of the Open Notebook. Every field has developed ways of cutting corners and ignoring some inconvenient data in order to simplify and get that paper out the door. In organic chemistry, people learn to take yields and parts of experimental descriptions less seriously after they have worked in a lab for a while trying to repeat those experiments and publish their own papers. It is not that researchers blatantly lie (though some do I am sure) but they don’t necessarily report ALL the facts. If a reaction was done three times and one of those times it failed completely, researchers typically just exclude the failed experiment and report the best yield. There could be some very useful information in that failed run from so many angles. By definition ALL those experiments must be in the research notebook of any decent experimentalist.

  2. pm286 says:

    (1) I agree.
    >What has become very interesting is the dialogue sparked by the Open Notebook/Open Source Science debate. You say the main point about publishing science is communicating it. But then we have seen from Heather Piwowar how difficult it is to get raw data from many researchers.
    Yes. The C-word is very useful. No-one can publicly admit to being against increased communication. So it becomes increasingly difficult to create technologies and protocols that protect closed interests.
    >I wonder how many journal articles can survive the scrutiny of the Open Notebook. Every field has developed ways of cutting corners and ignoring some inconvenient data in order to simplify and get that paper out the door. In organic chemistry, people learn to take yields and parts of experimental descriptions less seriously after they have worked in a lab for a while trying to repeat those experiments and publish their own papers. It is not that researchers blatantly lie (though some do I am sure) but they don’t necessarily report ALL the facts. If a reaction was done three times and one of those times it failed completely, researchers typically just exclude the failed experiment and report the best yield. There could be some very useful information in that failed run from so many angles. By definition ALL those experiments must be in the research notebook of any decent experimentalist.
    Yes. We are all human – an how many of us archive all our preprints as we are meant to, etc. We will probably have to see a mixture of mandates and better software (that’s of course where CML comes in). But, in the long run, the work has to be reproducible. CrystalEye and OSCAR show up errors and typos – no publishers can say “we don’t really care about errors” in public.
    So yes, Open Notebook Science (are you still happy with that term – I rather forced you into it 🙂 is enormously powerful. And there is a strong undercurrent of change. Too many fraud/mistakes – I heard that the Australian principals wanted all their stuff in repositories to protect against allegations. That sort of thing would be massive.

Leave a Reply

Your email address will not be published. Required fields are marked *