AAAI2022最新「贝叶斯优化」教程报告，230+页PPT阐述BO基础到高级主题

2022 年 3 月 8 日 专知

【导读】国际人工智能会议AAAI 2022论文将在全程线上举办，时间在 2 月 22 日-3 月 1 日，本届大会也将是第 36 届 AAAI 大会。大会涵盖了众多最近研究Tutorial报告，来自Jana Doppa等学者共同做了关于贝叶斯优化的进展报告，非常值得关注!

许多工程和科学应用，包括自动机器学习(例如，神经结构搜索和超参数调优)，都需要做出设计选择，以优化一个或多个昂贵的评估目标。一些例子包括调整编译器的旋钮，以优化一组软件程序的性能和效率; 设计新材料以优化强度、弹性和耐久性;并设计硬件优化性能，功率和面积。贝叶斯优化(BO)是一种有效的框架，用于解决函数求值昂贵的黑盒优化问题。BO的核心思想是利用真实的实验数据建立一个廉价的替代统计模型(如高斯过程);并利用它来智能地选择实验序列或使用采集函数的功能评估，例如期望改进(EI)和上置信度界限(UCB)。对于连续输入空间，在单保真度设置(即，实验昂贵且函数评估准确)的单目标优化中，有大量关于BO的工作。然而，BO近年来的工作已经集中在更具挑战性的问题设置，包括多目标优化;利用多保真度函数评估进行优化(不同的资源成本和评估的准确性);用黑盒约束优化应用到安全;组合空间的优化(例如，序列、树和图);混合空间(离散和连续输入变量的混合)的优化。本教程的目标是通过关注挑战、原则、算法思想和它们的连接，以及重要的现实世界应用，从基础到这些最新进展，呈现BO的全面调研。

目录内容：

0. Big Picture of Bayesian Optimization (BO) | Slides

Motivation from real-world applications
Challenges of optimizing expensive black-box functions
Overview of Bayesian optimization
Three key elements: statistical model, acquisition function (AF), and acquisition function optimizer (AFO)

1. Foundations of BO | Slides

Background on Gaussian Processes (GPs)

Two views of GPs: weight space and function space
Learning and inference with GPs
Scalability challenges and solutions

Background on Acquisition Functions (AFs)

Exploration vs. Exploitation trade-off
Example AFs: Expected improvement (EI), Upper confidence bound (UCB), Thompson Sampling (TS), and Information-theoretic methods
Optimizers: DIRECT, gradient-based methods, and evolutionary search

2. BO over Combinatorial Spaces | Slides

Motivating Applications

Manycore systems design
Biological sequence design
Drug/Vaccine/Molecule design

Key Challenges

Effective surrogate models for combinatorial structures (e.g., sequences, trees, graphs)
Trading-off complexity of statistical model and tractability of AFO

BO Methods over Original Space

Simple models and tractable AFO
Complex models and heuristic search
Complex models and tractable search
Complex models and effective search via learning-to-search

BO Methods over Latent Continuous Space

Advantages of reduction to BO over continuous space
Deep generative models for learning latent space
Challenges of BO over latent space: valid structures, high-dimensionality, imperfect decoder
Recent algorithmic advances: weighted re-training, LADDER, and high-dimensional BO

3. BO over Hybrid Spaces | Slides

Motivating Applications

Material design
Microbiome design
Auto ML tasks

Key Challenges

Effective surrogate models for hybrid structures
Trading-off complexity of statistical model and tractability of AFO

BO Methods over Original Space

Simple models and tractable AFO
Complex models and heuristic search

4. Multi-fidelity BO | Slides

Background on Multi-fidelity optimization

Function approximations (fidelities) with varying cost and accuracy of evaluation
Discrete fidelity vs. Continuous fidelity
Example real-world applications (e.g., Auto ML)