We consider the problem of minimizing a high-dimensional objective function, which may include a regularization term, using (possibly noisy) evaluations of the function. Such optimization is also called derivative-free, zeroth-order, or black-box optimization. We propose a new $\textbf{Z}$eroth-$\textbf{O}$rder $\textbf{R}$egularized $\textbf{O}$ptimization method, dubbed ZORO. When the underlying gradient is approximately sparse at an iterate, ZORO needs very few objective function evaluations to obtain a new iterate that decreases the objective function. We achieve this with an adaptive, randomized gradient estimator, followed by an inexact proximal-gradient scheme. Under a novel approximately sparse gradient assumption and various different convex settings, we show the (theoretical and empirical) convergence rate of ZORO is only logarithmically dependent on the problem dimension. Numerical experiments show that ZORO outperforms the existing methods with similar assumptions, on both synthetic and real datasets.
翻译:我们考虑尽量减少一个高维目标函数的问题,其中可能包括一个正规化术语,使用(可能吵闹的)对函数的评价。这种优化也称为无衍生物、零顺序或黑盒优化。我们提出一个新的$textbf ⁇ $$oroth-$textbf{O}$rder $textbf{R}$egalizionization $tlebbf{O}$ptifical 方法,称为ZORO。当基底梯度在一个迭代中大约稀少时,ZORO需要很少客观的函数评价来获得一个新的迭代来减少目标函数。我们用一个适应性、随机的梯度估计值来实现这一目标,然后用一个不精确的准偏差的准度估计公式。在一种新颖的近于稀疏渐渐变的假设和不同的同形设置下,我们展示ZORO的聚合率(理论和经验)仅仅取决于问题的维度。数字实验显示ZORO在合成数据和真实数据设置上都比现有方法的相似。