The Neural Testbed: Evaluating Predictive Distributions

Posterior predictive distributions quantify uncertainties ignored by point estimates. This paper introduces The Neural Testbed: an opensource library for controlled and principled evaluation of agents that generate such predictions. Crucially, agents are assessed not only on the quality of their marginal predictions per input, but also on their joint predictions across many inputs. We evaluate a range of agents using a neural-network-based data generating process. Our results indicate that some well-known agents that produce accurate marginal predictions do not fare well with joint predictions. We show that the quality of joint predictions drives performance in downstream decision tasks, and highlight the importance of this observation to the community.