Markov chain Monte Carlo (MCMC) algorithms have long been the main workhorses of Bayesian inference. Among them, Hamiltonian Monte Carlo (HMC) has recently become very popular due to its efficiency resulting from effective use of the gradients of the target distribution. In privacy-preserving machine learning, differential privacy (DP) has become the gold standard in ensuring that the privacy of data subjects is not violated. Existing DP MCMC algorithms either use random-walk proposals, or do not use the Metropolis--Hastings (MH) acceptance test to ensure convergence without decreasing their step size to zero. We present a DP variant of HMC using the MH acceptance test that builds on a recently proposed DP MCMC algorithm called the penalty algorithm, and adds noise to the gradient evaluations of HMC. We prove that the resulting algorithm converges to the correct distribution, and is ergodic. We compare DP-HMC with the existing penalty, DP-SGLD and DP-SGNHT algorithms, and find that DP-HMC has better or equal performance than the penalty algorithm, and performs more consistently than DP-SGLD or DP-SGNHT.
翻译:在保护隐私的机器学习中,差异隐私已成为确保数据主体隐私不受侵犯的黄金标准;现有的DP MMC算法要么使用随机行走建议,要么不使用大都会-成交(MH)接受测试,以确保趋同,同时又不将其步步数减为零;我们利用最近提议的DP MMC算法(即惩罚算法)提出HMC接受测试的DP变式,该变式以最近提议的DP MMC算法(即惩罚算法)为基础,增加了对HMC的梯度评价的噪音;我们证明由此产生的算法与数据主体的隐私权不相违背,而且具有ergodic。我们把DP-HMC算法与现有的惩罚(DP-SGLD和DP-SGNHT算法)进行比较,发现DP-HMC比惩罚算法更好或平等,并且比DP-SGLD或DPSG-DD/DPGDD-DGD/DPGGGD)更一致地执行。