Several works have proven that finetuning is an applicable approach for debiasing contextualized word embeddings. Similarly, discrete prompts with semantic meanings have shown to be effective in debiasing tasks. With unfixed mathematical representation at the token level, continuous prompts usually surpass discrete ones at providing a pre-trained language model (PLM) with additional task-specific information. Despite this, relatively few efforts have been made to debias PLMs by prompt tuning with continuous prompts compared to its discrete counterpart. Furthermore, for most debiasing methods that alter a PLM's original parameters, a major problem is the need to not only decrease the bias in the PLM but also to ensure that the PLM does not lose its representation ability. Finetuning methods typically have a hard time maintaining this balance, as they tend to violently remove meanings of attribute words. In this paper, we propose ADEPT, a method to debias PLMs using prompt tuning while maintaining the delicate balance between removing biases and ensuring representation ability. To achieve this, we propose a new training criterion inspired by manifold learning and equip it with an explicit debiasing term to optimize prompt tuning. In addition, we conduct several experiments with regard to the reliability, quality, and quantity of a previously proposed attribute training corpus in order to obtain a clearer prototype of a certain attribute, which indicates the attribute's position and relative distances to other words on the manifold. We evaluate ADEPT on several widely acknowledged debiasing benchmarks and downstream tasks, and find that it achieves competitive results while maintaining (and in some cases even improving) the PLM's representation ability. We further visualize words' correlation before and after debiasing a PLM, and give some possible explanations for the visible effects.
翻译:一些作品证明微调是用来降低背景化字嵌入内容的可适用方法。 同样,具有语义含义的离散提示也证明在贬低任务方面是有效的。由于象征性的不固定数学代表制,连续提示通常超过离散语言,提供经过预先训练的语言模式(PLM),并附加特定任务的信息。尽管如此,通过与离散的对应方进行连续调换,降低PLM(PLM)是一个可适用的方法。此外,对于改变PLM原有参数的多数贬低方法,一个主要问题是不仅需要降低PLM的偏差性,而且还需要确保PLM的代表性不丧失。 微调方法通常会超过离散的语言,因为提供事先经过训练的语言模式(PLM)和额外的信息。我们提出了ACTPT(ACTPT)方法,通过快速调换调来降低PLM(PM),同时在消除偏差和确保代表性能力之间保持微妙的平衡。为了实现这一点,我们提议了一个新的培训标准,我们提议在进行一系列的相对性学习和精确性实验后, 将某些属性提升其排序,然后调整。