Sturm, “W_hither the Design of Experiments in MIR?”
Precis: This position paper argues for developing a formal design of experiments (DOE) framework compatible with the kinds of experiments performed in MIR. I attempt to make clear the principal aim of DOE: to help one produce the most reliable evidence at the least cost. (What’s not to love about that?) I spend a majority of the paper dealing with counter arguments made against five of my failed grant proposals from the past 3 years. However, I do describe some of the difficulties in trying to shoehorn existing practices into an existing DOE framework. We are a long ways away from having a functional DOE framework, so I propose some alternative approaches to evaluation.
Rodríguez, Sturm and Maruri, “Analysing Scattering-based Music Content Analysis Systems: Where’s the Music?”
(Big ups to my PhD student Fran for accomplishing this within the first 6 months of starting!)
Precis: This paper in essense demonstrates how to perform system analysis in order to determine what a machine music listening system has actually learned to do. (In this way it is a nice partner for my paper above.) We analyse scattering-based feature systems trained and tested in the GTZAN dataset (extending our past work). We discover that it reproduces a large amount of GTZAN ground truth by exploiting information in subsonic frequencies, between 0-20 Hz. Whether or not this is an acceptable strategy depends on the use case of such a system; but at least we have revealed the “magic.”