Fairness Testing: A Comprehensive Survey and Analysis of Trends

Software systems are vulnerable to fairness bugs and frequently exhibit unfair behaviors, making software fairness an increasingly important concern for software engineers. Research has focused on helping software engineers to detect fairness bugs automatically. This paper provides a comprehensive survey of existing research on fairness testing. We collect 113 papers and organise them based on the testing workflow (i.e., the testing activities) and the testing components (i.e., where to find fairness bugs) for conducting fairness testing. We also analyze the research focus, trends, promising directions, as well as widely-adopted datasets and open source tools for fairness testing.