Following up on my previous experiments, let’s take a look at some results when
we train and test on hi-fi audio recordings.
As we saw before, with the Bergstra et al. method, recognition accuracy was quite high at just below 80%.
As far as I have seen in the literature, this is usually as far as people go when looking at the results.
Let’s dig a little deeper and see where mistakes are made.
Below we see the 10 classifications of all 100 files labeled Disco
from 10 runs of 5-fold cross validation. (The darker the square, the more times the exceprt was labeled that particular genre.)
Some things here are very encouraging. For instance, entry 27 should actually be labeled Hip hop since it is “Rapper’s Delight” by the Sugarhill Gang; and the algorithm seems to agree with 10 out of 10 votes there.
Entry 15 though is the quintessential Disco-defining tune “Boogie Nights” by Heatwave, and the algorithm has labeled it Pop.
Maybe that is ok.
However, not ok is 8 times out of 10, “Shake your booty” (entry 69) is labeled Blues.
And George McCrae’s “I can’t leave you alone” (entry 39) is 10 out of 10 times labeled Country.
Just to hear how Country this tune is, hear it for yourself:
We also see entry 64, “Funky Town” by Lipps, Inc., is 10 out of 10 Reggae.
“Funky Town” is only one of the greatest Disco hits ever.
Most damning though is entry 83, which 10 times out of 10 is classified as Rock.
Any algorithm able to distinguish Disco from the other nine categories would not make such a mistake:
That there I call the “Disco Duck Test”, and any good music genre algorithm should be able to get it from the bass line, horns, and open high hat pattern — except current algorithms do not look for such high-level things.
What about the method employing sparse approximation?
Here are its Disco entry confusions (done here with 100 tests of 10-fold cross validation).
Applying the Disco Duck Test, we are happy to see entry 83 has been correctly classified.
And “Shake your booty” (entry 69) is labeled Disco.
Furthermore, it classified LaToya Jackson’s mislabeled excerpts (entries 23 and 26) as Pop. And entry 85, “Wordy Rappinghood”, it thinks is Hip hop — which it is! Not Disco.
So there is hope!
However, “Rapper’s Delight” by the Sugarhill Gang, entry 27, is now 100 out of 100 Reggae.
“Boogie Nights” (entry 15) is more Pop than Disco.
And George McCrae’s “I can’t leave you alone” (entry 39) is Reggae and a little Pop, but never Disco.
Furthermore, and unforgivably, this algorithm considers
entry 67 as 100% Metal.
Here is the pure Metal for you:
(Entry 72 is also by Abba, and apparently pure Metal.)
There are many other problems like this in the other genres.
For instance, one of the methods thinks “The Lion Sleeps Tonight” and “A whole new world” from Aladdin are 10 out of 10 of the Blues genre. Beastie Boy’s “You gotta fight” is 10 out of 10 Metal (which is not as unforgivable as the other mistakes); AC/DC’s “Highway to hell” is nearly Disco, and so on and so forth.
Of course, we don’t want to throw the baby out with the bath water, but it is clear to me that the time is long past to answer whether the algorithms of the past decade of work even discriminate between music genres, or just between confounding variables.