We present new algorithms for optimizing non-smooth, non-convex stochastic objectives based on a novel analysis technique. This improves the current best-known complexity for finding a $(\delta,\epsilon)$-stationary point from $O(\epsilon^{-4}\delta^{-1})$ stochastic gradient queries to $O(\epsilon^{-3}\delta^{-1})$, which we also show to be optimal. Our primary technique is a reduction from non-smooth non-convex optimization to online learning, after which our results follow from standard regret bounds in online learning. For deterministic and second-order smooth objectives, applying more advanced optimistic online learning techniques enables a new complexity of $O(\epsilon^{-1.5}\delta^{-0.5})$. Our techniques also recover all optimal or best-known results for finding $\epsilon$ stationary points of smooth or second-order smooth objectives in both stochastic and deterministic settings.
翻译：我们提出基于新分析技术的优化非悬浮、非混凝土的随机目标的新算法。 这提高了目前最著名的复杂程度,从O(\\ delta,\ epsilon) 美元到 $(\ epsilon) 4 ⁇ delta ⁇ ⁇ -1} ($) 的随机梯度查询到 $(\ epsilon) 美元(\\ epsilon ⁇ 3 ⁇ delta ⁇ -1}) 美元。 我们的主要技术是将非悬浮的非凝固非凝固优化降低到在线学习, 之后, 我们的结果从在线学习的标准遗憾界限中得出。 对于确定性和第二顺序的平滑目标,应用更先进的乐观在线学习技术可以使$(\ epsilon ⁇ - 1.5 ⁇ delta ⁇ - 0.5} 美元产生新的复杂程度。 我们的技术还恢复了所有最佳或最著名的结果, 以寻找 $\ epsilon 平滑或第二等平坦目标的固定点, 。
相关内容
微信扫码咨询专知VIP会员