One of the challenges in machine learning research is to ensure that presented and published results are sound and reliable. Reproducibility, that is obtaining similar results as presented in a paper or talk, using the same code and data (when available), is a necessary step to verify the reliability of research findings. Reproducibility is also an important step to promote open and accessible research, thereby allowing the scientific community to quickly integrate new findings and convert ideas to practice. Reproducibility also promotes the use of robust experimental workflows, which potentially reduce unintentional errors. In 2019, the Neural Information Processing Systems (NeurIPS) conference, the premier international conference for research in machine learning, introduced a reproducibility program, designed to improve the standards across the community for how we conduct, communicate, and evaluate machine learning research. The program contained three components: a code submission policy, a community-wide reproducibility challenge, and the inclusion of the Machine Learning Reproducibility checklist as part of the paper submission process. In this paper, we describe each of these components, how it was deployed, as well as what we were able to learn from this initiative.
翻译:机器学习研究的挑战之一是确保所提出和公布的结果是健全和可靠的; 采用相同的代码和数据(如果有的话)进行复制是核实研究结果可靠性的必要步骤,这是利用相同的代码和数据(如果有的话)取得类似结果,这是核查研究结果可靠性的必要步骤; 复制也是促进开放和无障碍研究的一个重要步骤,从而使科学界能够迅速将新的研究结果综合起来,并将想法转化为实践; 复制还促进使用强大的实验工作流程,这有可能减少无意的错误; 2019年,神经信息处理系统(NeurIPS)会议,即机器学习研究的首届国际会议,推出了一个复制方案,目的是改进全社区如何进行、交流和评价机器学习研究的标准。 该方案包含三个组成部分:守则提交政策,全社区范围的复制挑战,以及将机器学习可减少清单纳入文件提交过程。 在本文中,我们描述了每个组成部分是如何部署的,以及我们能够从这一倡议中学到什么。