"If it didn't happen, why would I change my decision?": How Judges Respond to Counterfactual Explanations for the Public Safety Assessment

Many researchers and policymakers have expressed excitement about how algorithmic explanations may enable more fair and responsible decision-making. However, recent experimental studies have found that explanations do not always improve human use of algorithmic advice. In this study, we shed light on how people interpret and respond to counterfactual explanations (CFEs) -- an explanation that shows how a model's output changes with marginal changes to an input -- in the context of pretrial risk assessment instruments (PRAIs). We ran think-aloud trials with eight sitting US state court judges, providing them with recommendations from the PRAI as well as CFEs. At first, judges misinterpreted the counterfactuals as real -- rather than hypothetical -- changes to defendants. Once judges understood what the counterfactuals meant, they ignored them, stating they must make decisions based only on the actual defendant in question. They also expressed a mix of reasons for ignoring or following the advice of the PRAI. These results add to the literature on how people use algorithms and explanations in unexpected ways and the challenges associated with creating effective human-algorithm collaboration.