Nonsmooth nonconvex optimization problems broadly emerge in machine learning and business decision making, whereas two core challenges impede the development of efficient solution methods with finite-time convergence guarantee: the lack of computationally tractable optimality criterion and the lack of computationally powerful oracles. The contributions of this paper are two-fold. First, we establish the relationship between the celebrated Goldstein subdifferential~\citep{Goldstein-1977-Optimization} and uniform smoothing, thereby providing the basis and intuition for the design of gradient-free methods that guarantee the finite-time convergence to a set of Goldstein stationary points. Second, we propose the gradient-free method (GFM) and stochastic GFM for solving a class of nonsmooth nonconvex optimization problems and prove that both of them can return a $(\delta,\epsilon)$-Goldstein stationary point of a Lipschitz function $f$ at an expected convergence rate at $O(d^{3/2}\delta^{-1}\epsilon^{-4})$ where $d$ is the problem dimension. Two-phase versions of GFM and SGFM are also proposed and proven to achieve improved large-deviation results. Finally, we demonstrate the effectiveness of 2-SGFM on training ReLU neural networks with the \textsc{Minst} dataset.
翻译:在机器学习和商业决策中广泛出现非软性非软性优化问题,而两个核心挑战则阻碍以一定时间趋同的保证,开发高效解决方案方法,即:缺乏可计算可移动的最佳标准和缺乏计算能力强的神器。本文的贡献是双重的。首先,我们建立了著名的Goldstein succepteal-citep{Goldstein-1977-Oppimization}和统一平滑之间的关系,从而为设计无梯度方法提供了基础和直觉,这些方法可以保证在一定时间上与一套金斯坦固定点趋同。第二,我们建议采用无梯度方法和随机性GMM(GFM)方法来解决非软性非软性非软性非软性软体优化问题,并证明两者都能将利普西茨函数的$(delta,$-epsilton)-Goldstein Steptiary点还回回$form,以美元(d3/2 ⁇ - ⁇ -1 ⁇ -empsilon} 4} 。第二, 美元的拟议中度FM-FM-G-FM-G-G-SDFM-L-L-S-S-L-L-G-G-L-K-K-K-K-L-L-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-G-