Algorithmic encoding of protected characteristics in image-based models for disease detection

It has been rightfully emphasized that the use of AI for clinical decision making could amplify health disparities. A machine learning model may pick up undesirable correlations, for example, between a patient's racial identity and clinical outcome. Such correlations are often present in (historical) data used for model development. There has been an increase in studies reporting biases in image-based disease detection models. Besides the scarcity of data from underserved populations, very little is known about how these biases are encoded and how one may reduce or even remove disparate performance. There are concerns that an algorithm may recognize patient characteristics such as biological sex or racial identity, and then directly or indirectly use this information when making predictions. But it remains unclear how we can establish whether such information is actually used. This article aims to shed some light on these issues by exploring different methodology for assessing the inner working of disease detection models. We explore multitask learning and model inspection to assess the relationship between protected characteristics and prediction of disease. We believe our analysis framework could provide valuable insights in future studies in medical imaging AI. Our findings also call for further research to better understand the underlying causes of performance disparities.