Pitfalls and Potentials in Simulation Studies

Comparative simulation studies are workhorse tools for benchmarking statistical methods, but if not performed and reported transparently they may lead to overoptimistic or misleading conclusions. The current publication requirements adopted by statistics journals do not prevent questionable research practices such as selective reporting. There have been numerous suggestions and initiatives to improve on these issues but little progress can be seen to date. In this paper we discuss common questionable research practices which undermine the validity of findings from comparative simulation studies. To illustrate our point, we invent a novel prediction method with no expected performance gain and benchmark it in a pre-registered comparative simulation study. We show how easy it is to make the method appear superior over well-established competitor methods if no protocol is in place and various questionable research practices are employed. Finally, we provide researchers, reviewers, and other academic stakeholders concrete suggestions for improving the methodological quality of comparative simulation studies, most importantly the need for pre-registered simulation protocols.