Towards Instance-Wise Calibration: Local Amortized Diagnostics and Reshaping of Conditional Densities (LADaR)

There is a growing interest in conditional density estimation and generative modelling of a target $y$ given complex inputs $\mathbf{x}$. However, off-the-shelf methods often lack instance-wise calibration -- that is, for individual inputs $\mathbf{x}$, the individual estimated probabilities can be very different from the true probabilities, even when the estimates are reasonable when averaged over the entire population. This paper introduces the LADaR (Local Amortized Diagnostics and Reshaping of Conditional Densities) framework and proposes an algorithm called $\texttt{Cal-PIT}$ that produces interpretable local calibration diagnostics and includes a mechanism to recalibrate the initial model. Our $\texttt{Cal-PIT}$ algorithm learns a single local probability-probability map from calibration data to assess and quantify where corrections are needed across the feature space. When necessary, it reshapes the initial distribution into an estimate with approximate instance-wise calibration. We illustrate the LADaR framework by applying $\texttt{Cal-PIT}$ to synthetic examples, including probabilistic forecasting with sequences of images as inputs, akin to predicting the wind speed of tropical cyclones from satellite imagery. Our main science application is conditional density estimation of galaxy distances given imaging data (so-called photometric redshift estimation). On a benchmark photometric redshift data challenge, $\texttt{Cal-PIT}$ achieves better conditional density estimation (as measured by the conditional density estimation loss) than all 11 other literature methods tested. This demonstrates its potential for meeting the stringent photometric redshift requirements for next generation weak gravitational lensing analyses.