How reproducibility tipped the scale toward article acceptance

Today, I present a talk at the SoundSoftware 2014 Third Workshop on Software and Data for Audio and Music Research: “How reproducibility tipped the scale toward article acceptance“.
I discuss a recent episode in which our submission of a negative result article — contradicting previously published work — was favorably reviewed, and eventually published (here). The review process, and the persuasion of the reviewers, were greatly aided by our efforts at reproducibility. We won a reproducibility prize last year for this work.


2 thoughts on “How reproducibility tipped the scale toward article acceptance

  1. Are you kidding? Did a reviewer really say “This journal should defend research, not debunk it” when submitting that? Whoever that was, should go back to school and learn the principles of this job.


  2. Hi Julian. That is a synopsis. Here is the full review:
    Though I find this paper pretty interesting in the end, I’m afraid it is not fit for publication in this journal, aimed at defending new research.
    It would be a perfect fit however for a negative results oriented journal. There has been one issue back in 2004 of a Journal of Negative Results in Speech and Audio Sciences and I would suggest the authors of this paper to re-initiate such an initiative (for info, here is a list of negative results oriented publications in various domains ).
    In fact, even if the paper would be serious enough for publication in a journal, maybe it is more suited for a conference such as e.g. ISMIR because it would certainly create a passionate debate about whether it is worth spending some time writing papers about negative results and whether it is worth trying to clean up the field of weak/bad results. In this case, I would find relevant that the authors slightly rewrite their paper to place it in the context of negative results research and position themselves more explicitely in favor of publication of negative results (together with reproducibility of results).
    Anyway, in the end I pretty much agree with reviewer 3 that the original IGS paper is not that influential even though 10 papers referencing the original IGS paper are found.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s