Algorithmic encoding of protected characteristics and its implications on performance disparities

It has been rightfully emphasized that the use of AI for clinical decision making could amplify health disparities. A machine learning model may pick up undesirable correlations, for example, between a patient's racial identity and clinical outcome. Such correlations are often present in (historical) data used for model development. There has been an increase in studies reporting biases in disease detection models. Besides the scarcity of data from underserved populations, very little is known about how these biases are encoded and how one may reduce or even remove disparate performance. There are concerns that an algorithm may recognize patient characteristics such as biological sex or racial identity, and then directly or indirectly use this information when making predictions. But it remains unclear how we can establish whether such information is actually used. This article aims to shed some light on these issues by exploring methodology allowing intuitive inspections of the inner working of machine learning models for image-based detection of disease. We also investigate how to address performance disparities and find automatic threshold selection to be an effective yet questionable technique, resulting in models with comparable true and false positive rates across subgroups. Our findings call for further research to better understand the underlying causes of performance disparities.