I think tapping the intelligence and motivation of a globally networked community is excellent, but contributions must be vetted. Now that SPAM detection appears to be solved, we need troll detection.
Take this bowed violin “tone” ranked #1 by the human good-sounds community. It is clearly a poor sound in the context of Mozart, but perhaps a superior sound in the context of Berio. The former is contradicted by the automated scoring system, but is a troll at play here?