Surrogate algorithms such as Bayesian optimisation are especially designed for black-box optimisation problems with expensive objectives, such as hyperparameter tuning or simulation-based optimisation. In the literature, these algorithms are usually evaluated with synthetic benchmarks which are well established but have no expensive objective, and only on one or two real-life applications which vary wildly between papers. There is a clear lack of standardisation when it comes to benchmarking surrogate algorithms on real-life, expensive, black-box objective functions. This makes it very difficult to draw conclusions on the effect of algorithmic contributions. A new benchmark library, EXPObench, provides first steps towards such a standardisation. The library is used to provide an extensive comparison of six different surrogate algorithms on four expensive optimisation problems from different real-life applications. This has led to new insights regarding the relative importance of exploration, the evaluation time of the objective, and the used model. A further contribution is that we make the algorithms and benchmark problem instances publicly available, contributing to more uniform analysis of surrogate algorithms. Most importantly, we include the performance of the six algorithms on all evaluated problem instances. This results in a unique new dataset that lowers the bar for researching new methods as the number of expensive evaluations required for comparison is significantly reduced.
翻译:Bayesian 优化等代用算法特别针对目标昂贵的黑盒优化问题设计了黑盒优化问题,如超参数调制或模拟优化。在文献中,这些算法通常使用成熟但没有昂贵目标的合成基准来评价,并且只针对不同文件之间差别很大的一两个现实应用。当在现实生活、昂贵、黑盒客观功能方面将代用算法基准化时,明显缺乏标准化。这使得很难就算法贡献的效果得出结论。一个新的基准图书馆(EXPOBENCHE)为这种标准化提供了第一步。图书馆用来对不同现实生活应用中四种昂贵的替代算法问题进行广泛的比较。这导致对勘探的相对重要性、目标的评价时间以及使用的模式有了新的认识。进一步的贡献是,我们公开了算法和基准问题实例,有助于更统一的代用算法分析。最重要的是,我们把六种不同的代用代用法的替代算法的绩效作为新的比较,这是对新算法中最昂贵的数值评估。我们把六种新算法的数值作为新的比较。