Here is a nice quote I just found:
“Machine learning occupies a fortunate position that makes systematic experimentation easy and profitable. However, this does not mean empirical researchers should report gratuitous experiments any more than theoreticians should publish vacuous proofs. Whether they lead to positive or negative results, experiments are worthwhile only to the extent that they illuminate the nature of learning mechanisms and the reasons for their success or failure.”
from P. Langley, “Machine learning as an experimental science,” Machine Learning, vol. 3, pp. 5–8, 1988.