AMP Workshop on Reproducible Research

I am currently attending the AMP Workshop on Reproducible Research: Tools and Strategies for Scientific Computing, organized by Ian Mitchell, Randy Leveque and Victoria Stodden. This workshop is organized as one of the satellite workshops around ICIAM, the International Congress on Industrial and Applied Mathematics. It’s been a very interesting workshop already, with nice talks and tutorials about tools and policies. Some more about it probably later on this blog. I definitely already have some new input for the links pages on the rest of this website… The talks and presentations will also be posted online on the workshop website.

At the end of this workshop, there will also be a community forum discussion, to which I also really look forward.

Quiet times, busy times

It’s been quiet here over the past months… which is actually a sign for how busy real life has been over that period, and how lousy I have been at posting and updating here, and reacting on the little tidbits in my mailbox.

So let’s get things started and more active again over here.

Earlier this month I gave a talk on reproducible research at a mini-symposium “Store, Share and Cite” at TU Delft (The Netherlands). Together with the other technical universities in The Netherlands, they have created a data center for long-term storing of data sets. Great initiative. The mini-symposium was also very interesting, with talks also from other data centers and publishers. In my opinion too much of a library activity, and not enough ‘real’ researchers joining, but I think that’s always difficult. Hey, there were some, and I believe they were interested.

The decline effect

I just got pointed to a very interesting article in The New Yorker. It’s not entirely on reproducible research itself, but on the scientific method, which I believe is related enough to mention it here. The article is about how difficult it is to do ‘good’ science, and not to be tricked into believing what we want to believe.

J. Lehrer, The Truth Wears Off, The New Yorker, December 2010.

Open Source Software Competition at ACM Multimedia

It’s already quite some time ago, so it became highly time for a new post. This is definitely not by lack of interesting topics to blog about, it’s more related to a continued lack of time…

Anyway, I was reading last week about the Open Source Software Competition at ACM Multimedia this year. I didn’t know about it yet, but this seems to be a yearly event. Some more information about it is available here. In this competition, a short paper describing the software has to be submitted together with the entire open source software package. On the page, it is also clearly written that reviewers will make a reasonable attempt at testing and running the software, and submissions that don’t run will be rejected. I think this is a great event, and would like to congratulate the winners!

At the same time, I believe this also raises a question: Why is it that we have to create a separate competition for this, and do not have these same conditions (of being able to verify results) for regular conference or journal submissions? Is it too much to require both novelty (which I believe is less strictly checked in this competition), and (easy) reproducibility from paper submissions?

On reproducing someone’s results…

It’s a good idea to try to reproduce someone else’s results. But when doing so, it’s important to give credit to the original authors, by making appropriate citations.

It’s not a good idea to copy someone else’s article, only changing the author list, and submitting it to another conference/journal. I’m glad to see IEEE takes appropriate measures when such a thing happens. And I am still amazed it actually happened…

What is the best reproducible research?

What is best research practice in terms of reproducibility? At the recent workshop in As (Norway), I had a discussion with Marc-Oliver Gewaltig, similar to discussions I had earlier with some other colleagues as well. So I decided to put it up here. All feedback is welcome!

The discussion boils down to the following question: Is it better (in terms of reproducibility) to make code and data available online and allow users to repeat your experiments (or simulations as Marc-Oliver would call them) obtaining the same results, or to describe your theory (model in Marc-Oliver’s terminology) in sufficient detail that people can verify your results by re-implementing your experiments and verifying that they obtain the same thing?

I personally believe both approaches have their pros and cons. With the first one, a reader can download the related code and data, and very easily verify that he/she can obtain the same results as presented in the paper. If he wants to analyze things further, there is already a first implementation available to start analyzing, or to test on other data. However, that certainly doesn’t take away the need for a good and clear description in the paper!

With the second approach, one avoids the risk that a bug in the code giving those results is not caught by a reader reproducing the results, because he can just “double-click” to repeat the experiment. The second approach allows a thorough verification of the presented concept/theory, as the reader independently re-implements the work and checks the results. I believe certain standardization bodies like MPEG use this approach to make sure that descriptions are sufficiently precise.

Personally, I think the second approach is a better, more thorough approach in an ideal world. Currently, I prefer the first one, because most people won’t go into the depth of re-implementing things, and the first approach already gives those people something. Something more than just the paper, allowing to get their hands dirty on it. And “more interested readers” may still re-implement, or start analyzing the code in detail.