In non-private stochastic convex optimization, stochastic gradient methods converge much faster on interpolation problems -- problems where there exists a solution that simultaneously minimizes all of the sample losses -- than on non-interpolating ones; we show that generally similar improvements are impossible in the private setting. However, when the functions exhibit quadratic growth around the optimum, we show (near) exponential improvements in the private sample complexity. In particular, we propose an adaptive algorithm that improves the sample complexity to achieve expected error $\alpha$ from $\frac{d}{\varepsilon \sqrt{\alpha}}$ to $\frac{1}{\alpha^\rho} + \frac{d}{\varepsilon} \log\left(\frac{1}{\alpha}\right)$ for any fixed $\rho >0$, while retaining the standard minimax-optimal sample complexity for non-interpolation problems. We prove a lower bound that shows the dimension-dependent term is tight. Furthermore, we provide a superefficiency result which demonstrates the necessity of the polynomial term for adaptive algorithms: any algorithm that has a polylogarithmic sample complexity for interpolation problems cannot achieve the minimax-optimal rates for the family of non-interpolation problems.
翻译:在非私人的随机同流层优化中,随机梯度方法在内推问题 -- -- 存在同时尽量减少所有抽样损失的解决办法的问题 -- -- 而不是非内推问题 -- -- 中聚集得更快得多;我们表明,在私人环境中,一般相似的改进是不可能的。然而,当功能在最佳范围内呈现二次增长时,我们显示私人抽样复杂性的指数性改善(接近),同时为非内推问题保留标准的微量-最优抽样复杂性。我们证明,从美元到美元(frac){1-(alpha)_rho)+\(frac){d-(d) ⁇ pharepslon}\log\log\left(frac{1-alpha}right)在非内实现预期的误差($/rho>0)。我们建议一种适应性算法,即提高样本复杂性,从美元到美元(dol-alphal-al)美元到美元($1-alphal-pha)之间,我们证明一个较低约束的值显示尺寸依赖性术语的误差。此外,我们提供了一种超效率结果,对于任何多式的内算方法的复杂程度,不能实现任何多边的复杂度。