Replicability is not reproducibility: nor is it good science

  1. (PDF, 307 KB)
AuthorSearch for:
Proceedings titleProceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML, Montreal, Canada,2009
ConferenceEvaluation Methods for Machine Learning Workshop, the 26th ICML, June 14-18, 2009, Montreal, Canada
AbstractAt various machine learning conferences, atvarious times, there have been discussionsarising from the inability to replicate theexperimental results published in a paper.There seems to be a wide spread view that weneed to do something to address this prob-lem, as it is essential to the advancementof our field. The most compelling argumentwould seem to be that reproducibility of ex-perimental results is the hallmark of science.Therefore, given that most of us regard ma-chine learning as a scientific discipline, beingable to replicate experiments is paramount.I want to challenge this view by separatingthe notion of reproducibility, a generally de-sirable property, from replicability, its poorcousin. I claim there are important differ-ences between the two. Reproducibility re-quires changes; replicability avoids them. Al-though reproducibility is desirable, I contendthat the impoverished version, replicability,is one not worth having.
Publication date
AffiliationNational Research Council Canada; NRC Institute for Information Technology
Peer reviewedYes
NPARC number23002091
Export citationExport as RIS
Report a correctionReport a correction
Record identifier54187bb4-a8e2-4ce9-9067-9938dc403bea
Record created2017-08-10
Record modified2017-08-10
Bookmark and share
  • Share this page with Facebook (Opens in a new window)
  • Share this page with Twitter (Opens in a new window)
  • Share this page with Google+ (Opens in a new window)
  • Share this page with Delicious (Opens in a new window)