Regret Loss of Prejudiced Algorithms

The behavior of predictive algorithms built on data generated by a prejudiced human decision-maker is a prominent concern in the sphere of algorithmic bias. We consider the setting of a statistical and taste-based discriminator screening members of a disadvantaged group. We suppose one of two algorithms are used to score individuals: the algorithm $s_1$ favors disadvantaged individuals while the algorithm $s_2$ exemplifies the group-based prejudice in the training data set. Abstracting away from the estimation problem, we instead evaluate which of the two algorithms the discriminator prefers by using a version of regret loss generated by an algorithm. We define the notion of a regular and irregular environment and give theoretical guarantees on the firm's preferences in either case. Our main result shows that in a regular environment, greater levels of prejudice lead firms to prefer $s_2$ over $s_1$ on average. In particular, we prove the almost sure existence of a unique level of prejudice where a firm prefers $s_2$ over $s_1$ for any greater level of prejudice. Conversely, in irregular environments, the firm prefers $s_2$ for all $\tau$ almost surely.