Forecasting competitions are the equivalent of laboratory experimentation widely used in physical and life sciences. They provide useful, objective information to improve the theory and practice of forecasting, advancing the field, expanding its usage and enhancing its value to decision and policymakers. We describe ten design attributes to be considered when organizing forecasting competitions, taking into account trade-offs between optimal choices and practical concerns like costs, as well as the time and effort required to participate in them. Consequently, we map all major past competitions in respect to their design attributes, identifying similarities and differences between them, as well as design gaps, and making suggestions about the principles to be included in future competitions, putting a particular emphasis on learning as much as possible from their implementation in order to help improve forecasting accuracy and uncertainty. We discuss that the task of forecasting often presents a multitude of challenges that can be difficult to be captured in a single forecasting contest. To assess the caliber of a forecaster, we, therefore, propose that organizers of future competitions consider a multi-contest approach. We suggest the idea of a forecasting "athlon", where different challenges of varying characteristics take place.
翻译:预测竞赛相当于在物理科学和生命科学中广泛使用的实验室实验,它们提供了有用的客观信息,以改进预测理论和实践,推进实地,扩大利用范围,提高决策和决策人员的价值。我们描述了在组织预测竞赛时应考虑的十个设计属性,同时考虑到最佳选择与成本等实际关切之间的权衡,以及参与这些竞赛所需的时间和努力。因此,我们绘制了过去所有重大竞赛的设计属性图,查明它们之间的相似和差异,以及设计差距,并就未来竞赛所要包括的原则提出建议,特别强调从执行中学习,以便帮助改进预测准确性和不确定性。我们讨论,预测的任务往往带来许多挑战,难以在一次预测竞赛中反映出来。为了评估预测员的口吻,我们因此建议未来竞赛的组织者考虑多调和的办法。我们建议,在出现不同特点的不同挑战的地方,进行“预测”的想法。