Hyperparameter tuning is a common practice in the application of machine learning but is a typically ignored aspect in the literature on privacy-preserving machine learning due to its negative effect on the overall privacy parameter. In this paper, we aim to tackle this fundamental yet challenging problem by providing an effective hyperparameter tuning framework with differential privacy. The proposed method allows us to adopt a broader hyperparameter search space and even to perform a grid search over the whole space, since its privacy loss parameter is independent of the number of hyperparameter candidates. Interestingly, it instead correlates with the utility gained from hyperparameter searching, revealing an explicit and mandatory trade-off between privacy and utility. Theoretically, we show that its additional privacy loss bound incurred by hyperparameter tuning is upper-bounded by the squared root of the gained utility. However, we note that the additional privacy loss bound would empirically scale like a squared root of the logarithm of the utility term, benefiting from the design of doubling step.
翻译:超参数调试是应用机器学习的一种常见做法,但由于对总体隐私参数的负面作用,在关于隐私保护机学习的文献中,这是一个通常被忽视的方面。在本文中,我们的目标是通过提供有效的超参数调试框架,提供有效的超参数调控框架,提供不同的隐私,解决这个根本性但具有挑战性的问题。拟议方法使我们能够采用一个更广泛的超参数搜索空间,甚至在整个空间进行网格搜索,因为其隐私损失参数独立于超参数候选参数的数量。有趣的是,它与超参数搜索获得的效用相关,揭示了隐私和实用性之间的明确和强制性交换。理论上,我们表明,超参数调试测带来的额外隐私损失被获得的效用的平方根所覆盖。然而,我们注意到,额外的隐私损失将像实用术语的正方根一样,从双倍步骤的设计中受益。