Bayesian neural networks have successfully designed and optimized a robust neural network model in many application problems, including uncertainty quantification. However, with its recent success, information-theoretic understanding about the Bayesian neural network is still at an early stage. Mutual information is an example of an uncertainty measure in a Bayesian neural network to quantify epistemic uncertainty. Still, no analytic formula is known to describe it, one of the fundamental information measures to understand the Bayesian deep learning framework. In this paper, we derive the analytical formula of the mutual information between model parameters and the predictive output by leveraging the notion of the point process entropy. Then, as an application, we discuss the parameter estimation of the Dirichlet distribution and show its practical application in the active learning uncertainty measures by demonstrating that our analytical formula can improve the performance of active learning further in practice.
翻译:Bayesian神经网络在许多应用问题上成功地设计和优化了强大的神经网络模型,包括不确定性量化。然而,由于最近的成功,关于Bayesian神经网络的信息理论理解仍处于早期阶段。相互信息是Bayesian神经网络中一种不确定性计量方法的一个实例,以量化认知不确定性。然而,尚没有已知的解析公式来描述它,这是理解Bayesian深层学习框架的基本信息计量之一。在本文中,我们通过利用点过程昆虫的概念,在模型参数参数参数参数参数和预测输出之间得出了相互信息的分析公式。然后,作为一个应用,我们讨论了Drichlet分布的参数估计,并展示了其在积极学习不确定性计量中的实际应用,展示了我们的分析公式能够改善积极学习在实践中的绩效。