DFF FTP Independent Postdoc Project Results 2012-2013

Dear taxpayers of Denmark,

You have been very gracious to support my research the past two years by Independent postdoc grant No. 11-105218 from Det Frie Forskningsråd.
With the aid of this grant:
I have produced 11 journal article submissions, 3 of which are now published, 1 that has been accepted, and 2 that are currently in review;
I have produced 15 conference paper submissions, 12 of which are now published;
I have given 20 public seminars on my research around Europe and the USA;
and I have made 13 research visits of a week or more each.

Funny story: my DFF FTP postdoc grant proposal, titled, “Greedy Sparse Approximation and the Automatic Description of Audio and Music Data“,
is concerned with applying particular signal decomposition approaches to extracting content from audio data. However, very little of my resulting research directly addresses this goal. What I soon learned during the first quarter of my project is that the standard experimental framework practiced in a significant amount of research in this area is not as useful or illuminating as thought by its research community: invalid conclusions are being derived from poor evaluation using faulty data, which in turn can lead research widely astray, and waste valuable resources. The second quarter of my project led me to produce a thorough survey of the extent to which this is happening, and reveals a surprising lack of scientific practice in more than 500 published works. If the standard approach to evaluating a solution is faulty, then why continue to produce solutions and not change the evaluation? Hence, by the end of the first half of my project, it made no sense to produce new approaches to extracting information from audio signals and use the same poor evaluation to “measure” progress. Clearly, the most significant problem, and the one that must be addressed before all others, is how to scientifically evaluate such solutions. To this end, I devoted the last half of my project to answering why this has happened, and what can be done about it. As a result, many new directions have emerged that I will explore over the next several years with the various collaborators I have made during my DFF FTP Independent Postdoc.

Below the fold, I provide a detailed record of the results of my DFF FTP Independent Postdoc, both failures and successes.

Journal submissions:

1. B. L. Sturm, “When ‘Exact Recovery’ is Exact Recovery in Compressed Sensing,” submitted Jan. 2012 to IEEE Transactions on Signal Processing. (rejected)
2. B. L. Sturm, “The Tzanetakis music genre dataset: Its faults and the challenges they provide,” submitted Jan. 2012 to IEEE Transactions on Audio, Speech, and Language Processing. (rejected)
3. B. L. Sturm, “The GTZAN dataset: Its contents, faults, and their effects on music genre recognition evaluation,” submitted Jan. 2013 to IEEE Transactions on Audio, Speech, and Language Processing. (rejected)
4. B. L. Sturm, “The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future use,” submitted May 2013 to IEEE Transactions on Audio, Speech, and Language Processing. (rejected, becomes arXiv technical report 1306.1461)
5. B. L. Sturm, “Clever Hans is alive and well in MIR, and he is not just hiding in music genre recognition,” submitted May 2013 to IEEE Signal Processing Magazine. (rejected)
6. B. L. Sturm, B. Mailhé, and M. D. Plumbley, “On theorem 10 in ‘On polar polytopes and the recovery of sparse representations’,” IEEE Transactions on Information Theory, vol. 59, pp. 5206-5209, Aug. 2013.
7. B. L. Sturm and F. Gouyon, “Revisiting inter-genre similarity,” IEEE Signal Processing Letters, vol. 20, pp. 1050-1053, Nov. 2013.
8. B. L. Sturm, “Classification accuracy is not enough: On the evaluation of music genre recognition systems,” Journal of Intelligent Information Systems, vol. 41, no. 3, pp. 371-406, Dec. 2013.
9. B. L. Sturm, “The state of the art ten years after a state of the art: Future research in music information retrieval,” submitted Dec. 2013 to Journal of New Music Research. (accepted for publication)
10. B. L. Sturm, “A simple method to determine if a music information retrieval system is a horse,” submitted Dec. 2013 to IEEE Transactions on Multimedia. (in review)
11. F. Gouyon, B. L. Sturm, J. L. Oliveira, N. Hespanhol, and T. Langlois, “On evaluation in music auto-tagging research,” submitted Dec. 2013 to IEEE Transactions on Audio, Speech, and Language Processing. (in review)

Grant submissions

1. FP7-ICT-2011C FET OPEN STREP Young Explorers (€1,2 million) submitted Sep. 10, 2012. (not awarded)
2. Villum Fondens Young Investigator Programme, submitted July 2013. (not awarded)
3. Programme IFD — Science 2013 for French-Danish scientific co-operation (€3.000) submitted Nov. 2012 (awarded, to commence Feb. 2014)
4. Two grants from Center for Digital Music, Queen Mary University of London, June 2012, 2013 (awarded).

Conference submissions:

1. B. L. Sturm, “A Study on Sparse Vector Distributions and Recovery from Compressed Sensing,” submitted Apr. 2012 to the European Signal Processing Conference (rejected)
2. B. L. Sturm, “The Tzanetakis music genre dataset: Its faults and the challenges they provide,” submitted Apr. 2012 to the International Society for Music Information Retrieval. (rejected)
3. B. L. Sturm, “Three revealing experiments in music genre recognition,” submitted Apr. 2012 to the International Society for Music Information Retrieval. (rejected)
4. B. L. Sturm and P. Noorzad, “On Automatic Music Genre Recognition by Sparse Representation Classification using Auditory Temporal Modulations,” in Proceedings of Computer Music Modeling and Retrieval, pp. 379-394, June 2012.
5. B. L. Sturm, “When ‘Exact Recovery’ is Exact Recovery in Compressed Sensing,” in Proceedings of the European Signal Processing Conference, pp. 979-983, Aug. 2012.
6. P. Noorzad and B. L. Sturm, “Regression with sparse approximations of data,” in Proceedings of the European Signal Processing Conference, pp. 674-678, Aug. 2012.
7. B. L. Sturm and M. G. Christensen, “Comparison of Orthogonal Matching Pursuit Implementations,” in Proceedings of the European Signal Processing Conference, pp. 220-224, Aug. 2012.
8. B. L. Sturm, “A Survey of Evaluation in Music Genre Recognition,” in Proceedings of the 10th international workshop on Adaptive Multimedia Retrieval, Oct. 2012.
9. B. L. Sturm, “An Analysis of the GTZAN Music Genre Dataset“, in Proceedings of ACM Multimedia: Workshop on Music Information Retrieval with User-Centered and Multimodal Strategies, Nov. 2012.
10. B. L. Sturm, “Two Systems for Automatic Music Genre Recognition: What Are They Really Recognizing?“, Proceedings of ACM Multimedia: Workshop on Music Information Retrieval with User-Centered and Multimodal Strategies, Nov. 2012.
11. B. Mailhé, B. Sturm, and M. Plumbley, “Behavior of greedy sparse representation algorithms on nested supports,” in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 5710-5714, May 2013.
12. B. L. Sturm “Music genre recognition with risk and rejection,” in Proceedings of the IEEE International Conference on Multimedia & Expo, July 2013.
13. B. L. Sturm “On music genre classification via compressive sampling,” in Proceedings of the IEEE International Conference on Multimedia & Expo, July 2013.
14. B. L. Sturm “Evaluating music emotion recognition: Lessons from music genre recognition?,” in Proceedings of the IEEE International Conference on Multimedia & Expo, July 2013.
15. B. L. Sturm, “Formalizing evaluation in music information retrieval: A look at the MIREX automatic mood classification task,” in Proceedings of Computer Music Modeling and Retrieval, pp. 379-394, Oct. 2013.

Awards:

1. July 2013: Sound Software Reproducible Research Prize for journal submission

Technical reports:

1. B. L. Sturm, “The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future use,” http://arxiv.org/abs/1306.1461, June 2013.

Book reviews:

1. B. L. Sturm, “Book reivew: ‘An Introduction to Audio Content Analysis: Applications in Signal Processing and Music Informatics’ by Alexander Lerch, Computer Music Journal, vol. 37, no. 4, 2013.

Public seminars:

1. Departement Elektrotechniek, Catholic University Leuven, Belgium (March 21, 2012)
2. Intelligent Signal Processing Group, Technical University of Denmark (May 14, 2012)
3. Speech and Audio Processing Group, Imperial University, UK (May 30, 2012)
4. Centre for Digital Music of Queen Mary University of London, UK (May 23, 2012)
5. Sound Software Workshop, Queen Mary University of London, UK (June 18, 2012)
6. Centre for Mathematical Sciences, Lund University, Sweden (September 14, 2012)
7. Institut for Kommunikation, Aalborg University, Denmark (Oct. 29, 2012)
8. Communication Theory Lab, KTH-Royal Institute of Technology, Sweden (Dec. 3, 2012)
9. Austrian Research Institute for Artificial Intelligence, Vienna, Austria (Feb. 14, 2013)
10. Department of Computational Perception, University of Linz, Austria (Feb. 20, 2013)
11. L’IRISA (Institut de recherche en informatique et systèmes aléatoires), Rennes, France (Mar. 15, 2013)
12. Department of Informatics & Telecommunications, University of Athens, Greece (May 14, 2013)
13. Department of Informatics & Telecommunications, University of Athens, Greece (May 15, 2013)
14. Centre for Digital Music of Queen Mary University of London, UK (June 12, 2013)
15. Media Arts and Technology, University of California, Santa Barbara, USA (July 23, 2013)
16. Qarma Team, Aix-Marseille Université, Marseille, France (Oct. 18, 2013)
17. Austrian Research Institute for Artificial Intelligence, Vienna, Austria (Oct. 31, 2013)
18. Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain (Nov. 13, 2013)
19. Signal and Image Processing Department, Télécom ParisTech, Paris, France (Nov. 25, 2013)
20. Fraunhofer IAIS, Sankt Augustine, Germany (Dec. 4, 2013)

Research visits:

1. Departement Elektrotechniek, Catholic University Leuven, Belgium (Mar. 20-23, 2012)
2. Centre for Digital Music of Queen Mary University of London, UK (Apr. 16–June 28, 2012) Funded in part by EPSRC Platform Grant EP/E045235/1 at the Centre for Digital Music of Queen Mary University of London
3. Communication Theory Lab, KTH-Royal Institute of Technology, Sweden (Dec. 3-7, 2012)
4. Centre for Mathematical Sciences, Lund University, Sweden (Dec. 11-14, 2012)
5. Institute for Systems and Computer Engineering, INESC Porto, Portugal (Jan. 17-31, 2013)
6. Austrian Research Institute for Artificial Intelligence, Vienna, Austria (Feb. 13-28, 2013)
7. L’IRISA (Institut de recherche en informatique et systèmes aléatoires), Rennes, France (Mar. 10-23, 2013)
8. Institute for Systems and Computer Engineering, INESC Porto, Portugal (Apr. 8-25, 2013)
9. Department of Informatics & Telecommunications, University of Athens, Greece (Apr. 29-May 24, 2013)
10. Centre for Digital Music of Queen Mary University of London, UK (June 3-22, 2013) Funded in part by EPSRC Platform Grant EP/E045235/1 at the Centre for Digital Music of Queen Mary University of London.
11. Nick Collins, University of Sussex, UK (June 24-28, 2013)
12. Media Arts and Technology, University of California, Santa Barbara, USA (July 22-Aug. 2, 2013)
13. Fraunhofer IAIS, Sankt Augustine, Germany (Dec. 2-12, 2013)

Acknowledgments:
My wife Carla helped me to degrees beyond that which any words can describe. You do deserve co-authorship! “Thank you” to my many colleagues at AAU who gladly took up the slack resulting from my vacancy. “Thank you” to the many institutions and colleagues that I visited. “Thank you” to the Danish tax payers for making this program a possibility!