Comparative simulation studies are workhorse tools for benchmarking statistical methods. As with other empirical studies, the success of simulation studies hinges on the quality of their design, execution and reporting. If not conducted carefully and transparently, their conclusions may be misleading. In this paper we discuss various questionable research practices which may impact the validity of simulation studies, some of which cannot be detected or prevented by the current publication process in statistics journals. To illustrate our point, we invent a novel prediction method with no expected performance gain and benchmark it in a pre-registered comparative simulation study. We show how easy it is to make the method appear superior over well-established competitor methods if questionable research practices are employed. Finally, we provide concrete suggestions for researchers, reviewers and other academic stakeholders for improving the methodological quality of comparative simulation studies, such as pre-registering simulation protocols, incentivizing neutral simulation studies and code and data sharing.
翻译:与其他经验研究一样,模拟研究的成功取决于其设计、执行和报告的质量,如果不认真和透明地进行,其结论可能会产生误导。在本文件中,我们讨论了可能影响模拟研究有效性的各种有疑问的研究做法,其中一些做法无法被统计学期刊目前出版过程所发现或阻止。为了说明我们的观点,我们发明了一种没有预期业绩收益的新预测方法,并在预先登记的比较模拟研究中加以基准。我们证明,如果采用有疑问的研究做法,使该方法看起来优于既定的竞争者方法是多么容易。最后,我们向研究人员、审查人员和其他学术利益攸关方提出具体建议,以提高比较模拟研究的方法质量,如预先登记模拟协议,激励中立模拟研究以及代码和数据共享。</s>