Monthly Archives: April 2009

Plan for merging .org and .net sites

1 Reply

Patrick Vandewalle and I will be combining our efforts to develop a web site to promote reproducible research. He has the domain name reproducibleresearch.net while I have reproducibleresearch.org. His site is better than the one I’ve developed, so I’d rather support his effort than continue  my own.

I plan to leave this web site up for a few more weeks and then hand the .org name over to Patrick. During that time, some of the content from this site will be merged into the framework of his new site. Please go over to the new site and participate in the forums.

I plan continue blogging about reproducible research from time to time, but future posts will be on my personal blog, The Endeavour. I may write a few more posts here regarding the status of the transition.

Reproducible Research in Signal Processing – What, why, and how

2 Replies

I am glad to let you know that our paper has been published in the latest issue of IEEE Signal Processing Magazine:

P. Vandewalle, J. Kovacevic and M. Vetterli, Reproducible Research in Signal Processing – What, why, and how, IEEE Signal Processing Magazine, Vol. 26, Nr. 3, pp. 37-47, 2009, DOI: 10.1109/MSP.2009.932122.

Have you ever tried to reproduce the results presented in a research paper? For many of our current publications, this would unfortunately be a challenging task. For a computational algorithm, details such as the exact data set, initialization or termination procedures, and precise parameter values are often omitted in the publication for various reasons, such as a lack of space, a lack of self-discipline, or an apparent lack of interest to the readers, to name a few. This makes it difficult, if not impossible, for someone else to obtain the same results. In our experience, it is often even worse as even we are not always able to reproduce our own experiments, making it difficult to answer questions from colleagues about details. Following are some examples of e-mails we have received: “I just read your paper X. It is very completely described, however I am confused by Y. Could you provide the implementation code to me for reference if possible?” “Hi! I am also working on a project related to X. I have implemented your algorithm but cannot get the same results as described in your paper. Which values should I use for parameters Y and Z?”

Enjoy reading! And feel free to post your comments!

A sobering experience

Leave a reply

Last month, a few former colleagues at LCAV did some cross-testing of the reproducible research compendia available at rr.epfl.ch. And I must say, from the results I have seen so far, it is quite a sobering experience. Many of those which I considered to be definitely reproducible didn’t pass the test (entirely). I guess that shows again how difficult it is to make work really reproducible, even if you fully intend to do it. So that also leads me to my conviction that for papers that do not have code and data online, it is almost impossible to reproduce the exact results. There is work to be done on the road to reproducible research!

I’ll need to look further into the reasons why even some of my own work did not pass the test.