Deep Neural Network (DNN) models have been shown to have high empirical privacy leakages. Clinical language models (CLMs) trained on clinical data have been used to improve performance in biomedical natural language processing tasks. In this work, we investigate the risks of training-data leakage through white-box or black-box access to CLMs. We design and employ membership inference attacks to estimate the empirical privacy leaks for model architectures like BERT and GPT2. We show that membership inference attacks on CLMs lead to non-trivial privacy leakages of up to 7%. Our results show that smaller models have lower empirical privacy leakages than larger ones, and masked LMs have lower leakages than auto-regressive LMs. We further show that differentially private CLMs can have improved model utility on clinical domain while ensuring low empirical privacy leakage. Lastly, we also study the effects of group-level membership inference and disease rarity on CLM privacy leakages.
翻译:深海神经网络(DNN)模型被证明具有高度的经验隐私渗漏。临床语言模型(CLMs)在临床数据方面受过培训,用于改善生物医学自然语言处理任务的业绩。在这项工作中,我们调查通过白箱或黑箱进入CLMs进行的培训数据渗漏的风险。我们设计并使用会员推论攻击来估计BERT和GPT2等模型结构的经验隐私渗漏。我们显示,对CLMs的会员推论攻击导致非三边隐私渗漏高达7%。我们的结果显示,比较大的模型更小的经验隐私渗漏,而蒙蔽LMs的渗漏比自动递增LMs要少。我们进一步表明,差别化的私人CLMs可以在确保低经验隐私渗漏的同时改进临床领域的模型实用性。我们还研究了集体成员级别的推断和疾病对CLM隐私渗漏的影响。