Evaluating fairness can be challenging in practice because the sensitive attributes of data are often inaccessible due to privacy constraints. The go-to approach that the industry frequently adopts is using off-the-shelf proxy models to predict the missing sensitive attributes, e.g. Meta [Alao et al., 2021] and Twitter [Belli et al., 2022]. Despite its popularity, there are three important questions unanswered: (1) Is directly using proxies efficacious in measuring fairness? (2) If not, is it possible to accurately evaluate fairness using proxies only? (3) Given the ethical controversy over inferring user private information, is it possible to only use weak (i.e. inaccurate) proxies in order to protect privacy? Our theoretical analyses show that directly using proxy models can give a false sense of (un)fairness. Second, we develop an algorithm that is able to measure fairness (provably) accurately with only three properly identified proxies. Third, we show that our algorithm allows the use of only weak proxies (e.g. with only 68.85%accuracy on COMPAS), adding an extra layer of protection on user privacy. Experiments validate our theoretical analyses and show our algorithm can effectively measure and mitigate bias. Our results imply a set of practical guidelines for practitioners on how to use proxies properly. Code is available at github.com/UCSC-REAL/fair-eval.
翻译:评估公平性在实践中可能具有挑战性,因为由于隐私限制,数据的敏感属性往往无法获取,因此评估公平性在实践中可能具有挑战性。业界经常采用的方法是使用现成替代模型来预测缺失的敏感属性,例如Meta[Ala 等人,2021]和Twitter[Belli等人,2022]。尽管受到欢迎,但有三个重要问题没有得到解决:(1) 直接使用代理人是否有效衡量公平性?(2) 如果不是,能否仅使用代理人来准确评价公平性?(3) 鉴于在推断用户私人信息方面存在的道德争议,能否为保护隐私而仅使用现成替代模型?我们的理论分析表明,直接使用代理模型可以产生虚假的(不公)公平感。第二,我们开发一种算法能够(可能)准确衡量公平性,只有三个正确识别的代理人。如果不是,那么我们的算法允许仅使用薄弱的代理人(例如,在COMAS上只有68.85%的准确性),是否可能仅仅使用虚弱的替代模型? 为保护隐私性?我们的用户隐私性标准,如何在现实性分析中正确使用我们的理论性分析中能显示我们的精确性标准?