Yet another call for better empirical practice in computer science: C. T. Morrison and R. T. Snodgrass, “Viewpoint: Computer science can use more science,” Commun. ACM, vol. 54, pp. 36–38, June 2011.
I say “yet another call” because it is certainly not new. It was recognised 36 years ago: D. D. McCracken, P. J. Denning, and D. H. Brandin, “An ACM executive committee position on the crisis in experimental computer science,” Communications of the ACM, vol. 22, no. 9, pp. 503–504, 1979.
This motivated definition and examples of good experimental practice soon after:
P. J. Denning, “What is experimental computer science?,” Communications of the ACM, vol. 23, pp. 543– 544, Oct. 1980.
P. J. Denning, “Performance analysis: experimental computer science as its best,” Communications of the ACM, vol. 24, pp. 725–727, Nov. 1981.
A tutorial on proper experimental methods appeared:
V. Basili, R. Selby, and D. Hutchens, “Experimentation in software engineering,” IEEE Trans. Software Eng., vol. 12, no. 7, pp. 733–743, 1986.
Eight years after that, Fenton et al. observed that much much more work needs to be done:
N. Fenton, S. L. Pfleeger, and R. L. Glass, “Science and substance: A challenge to software engineers,” IEEE Software, vol. 11, pp. 86–95, July 1994.
Twelve years after that, Feitelson observed that much much more work needs to be done:
D. G. Feitelson, “Experimental computer science: The need for a cultural change,” tech. rep., The Hebrew University of Jerusalem, Dec. 2006.
The debate about whether computer science is science is settled (P. J. Denning, “Viewpoint: The science in computer science,” Commun. ACM, vol. 56, pp. 35–38, May 2013). The health of scientific practice in computer science, however, remains a point for serious debate after at least 36 years. In Morrison and Snodgrass’s “Viewpoint”, they highlight the scientific aspects of an everyday practice: debugging. “Programmers frequently demonstrate they are highly skilled at understanding complex interactions by forming models and testing and revising them.” Since the focus of this practice is on the instance and does not aim for generalisation, it is not science; but it does show some important aspects of scientific practice that are well-known in computer science. Morrison and Snodgrass provide a great success story of science in computer science, the “Theory of Locality”, which predicts “human-constructed information processing systems will exhibit locality.” I don’t fully understand what that is, but it has clearly been developed empirically with an interest on the general.
Morrison and Snodgrass argue that computer science needs scientific models to progress, and current skill sets can be leveraged to attain that goal: “We can adapt our already very skilled hypothesis testing in debugging and broaden it by asking more general questions, identifying the classes of system properties that contribute to behavior, identifying their boundary conditions, and working to fit them into a unified picture. We can also more broadly adopt additional methodological tools, such as from statistics and dynamic systems; a number of computer science subfields already profitably do so.”
I wonder how many more of these viewpoints and tutorials will appear before scientific practice in computer science reaches the state Morrison and Snodgrass are arguing for.