In this paper, we consider a theoretical model for injecting data bias, namely, under-representation and label bias (Blum & Stangl, 2019). We theoretically and empirically study its effect on the accuracy and fairness of fair classifiers. Theoretically, we prove that the Bayes optimal group-aware fair classifier on the original data distribution can be recovered by simply minimizing a carefully chosen reweighed loss on the bias-injected distribution. Through extensive experiments on both synthetic and real-world datasets (e.g., Adult, German Credit, Bank Marketing, COMPAS), we empirically audit pre-, in-, and post-processing fair classifiers from standard fairness toolkits for their fairness and accuracy by injecting varying amounts of under-representation and label bias in their training data (but not the test data). Our main observations are: (1) The fairness and accuracy of many standard fair classifiers degrade severely as the bias injected in their training data increases, (2) A simple logistic regression model trained on the right data can often outperform, in both accuracy and fairness, most fair classifiers trained on biased training data, and (3) A few, simple fairness techniques (e.g., reweighing, exponentiated gradients) seem to offer stable accuracy and fairness guarantees even when their training data is injected with under-representation and label bias. Our experiments also show how to integrate a measure of data bias risk in the existing fairness dashboards for real-world deployments
翻译:在本文中,我们考虑的是注射数据偏差的理论模型,即代表性不足和标签偏差(Blum & Stangl, 2019年)。我们从理论上和经验上研究其对公平分类者的准确性和公平性的影响。理论上,我们证明,在原始数据分配方面,贝耶斯最佳群体认识公平分类者对原始数据分配的最佳分类者可以通过简单尽量减少精心选择的偏差分布的重度损失来恢复。通过对合成和真实世界数据集(如成人、德国信用、银行销售、COMPAS)的广泛实验,我们从标准公平工具中,从公平性和准确性的角度,对处理前、内部和后公平性分类者进行实证性审计。我们的主要观察是:(1) 许多标准公平分类者的公平性和准确性随着其培训数据偏差的增加而严重降低。(2) 有关正确数据的简单物流回归模型往往在准确性和公平性方面优于准确性和公正性,对有偏差性的培训最公平的分类员对公平性数据进行了经验性分析。(3) 以简单、公平性的方法显示其真实性、准确性,在不可靠程度上也显示其真实性、真实性、可靠程度的标签上的数据。