The course is fully booked, with 23 students and a few auditors. We have a very good crop of papers for this inaugural edition of my PhD course. Some of these are classic papers (bold). Some are very new ones (italic). All deserve to be read and critically examined!

Nov. 1: Questions of Ethics

J. Bryson and A. Winfield, “Standardizing ethical design for artificial intelligence and autonomous systems,” Computer, vol. 50, pp. 116–119, May 2017.

Nov. 13: Questions of Performance

E. Law, “The problem of accuracy as an evaluation criterion,” in Proc. Int. Conf. Machine Learning: Workshop on Evaluation Methods for Machine Learning, 2008.

F. M.-Plumed, R. B. C. Prudêncio, A. M.-Usó, and J. H.-Orallo, “Making sense of item response theory in machine learning,” in Proc. ECAI, 2016.

Nov. 15: Questions of Learning

**D. J. Hand, “Classifier technology and the illusion of progress,” Statistical Science, vol. 21, no. 1, pp. 1–15, 2006.**

E. R. Dougherty and L. A. Dalton, “Scientific knowledge is possible with small-sample classification,” EURASIP J. Bioinformatics and Systems Biology, vol. 2013:10, 2013

Nov. 20: Questions of Sanity

I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in Proc. ICLR, 2015

*S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek & K.-R. Müller, “Unmasking Clever Hans predictors and assessing what machines really learn” Nature 2019*

Nov. 22: Questions of Statistics

**C. Drummond and N. Japkowicz, “Warning: Statistical benchmarking is addictive. kicking the habit in machine learning,” J. Experimental Theoretical Artificial Intell., vol. 22, pp. 67–80, 2010.**

*S. Makridakis, E. Spiliotis, V. Assimakopoulos, “Statistical and Machine Learning forecasting methods: Concerns and ways forward“, PLOS ONE 2018.*

**S. Goodman, “A Dirty Dozen: Twelve P-Value Misconceptions”, Seminars in Hematology, 2008.**

Nov. 27: Questions of Experimental Design

C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. Roth, “The reusable holdout: Preserving validity in adaptive data analysis,” Science, vol. 349, no. 6248, pp. 636–638, 2015.

T. Hothorn, F. Leisch, A. Zeileis, and K. Hornik, “The design and analysis of benchmark experiments,” Journal of Computational and Graphical Statistics, vol. 14, no. 3, pp. 675–699, 2005.

Nov. 29: Questions of Data

M. J. Eugster, F. Leisch, and C. Strobl, “(Psycho-)analysis of benchmark experiments: A formal frame- work for investigating the relationship between data sets and learning algorithms,” Computational Statistics & Data Analysis, vol. 71, pp. 986 – 1000, 2014.

*Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, Christopher Ré, “Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging” arXiv 2019*

S. Tolan, “Fair and unbiased algorithmic decision making: Current state and future challenges,” JRC Technical Reports, JRC Digital Economy Working Paper 2018-10, arXiv 2018.

Dec. 4: Questions of Sabotage

J. Su, D. V. Vargas, and K. Sakurai, “One pixel attack for fooling deep neural networks,” arXiv, vol. 1710.08864, 2017

*S. G. Finlayson, J. D. Bowers, J. Ito, J. L. Zittrain, A. L. Beam, I. S. Kohane, “Adversarial attacks on medical machine learning” Science 2019.*

Dec. 6: Questions of Interpretability

Z. Lipton, “The mythos of model interpretability,” in Proc. ICML Workshop on Human Interpretability in Machine Learning, 2016

*Malvina Nissim, Rik van Noord, Rob van der Goot, “Fair is Better than Sensational: Man is to Doctor as Woman is to Doctor” arXiv 2019.*

Dec. 11: Questions of Methodology

*Z. C. Lipton and J. Steinhardt, “Troubling trends in machine learning scholarship,” in Proc. ICML, 2018.*

Meyer, Michelle N., “Two Cheers for Corporate Experimentation: The A/B Illusion and the Virtues of Data-Driven Innovation” 13 Colo. Tech. L.J. 273 (2015).

Dec. 13: Questions of Application

**K. L. Wagstaff, “Machine learning that matters,” in Proc. Int. Conf. Machine Learning, pp. 529–536, 2012.**

*Cynthia Rudin, David Carlson, “The Secrets of Machine Learning: Ten Things You Wish You Had Known Earlier to be More Effective at Data Analysis” arXiv 2019.*

M. Fernández-Delgado, E. Cernadas, S. Barro, and D. Amorim, “Do we need hundreds of classifiers to solve real world classification problems?,” Journal of Machine Learning Research, vol. 15, pp. 3133–3181, 2014