Bayesian neural networks have successfully designed and optimized a robust neural network model in many application problems, including uncertainty quantification. However, with its recent success, information-theoretic understanding about the Bayesian neural network is still at an early stage. Mutual information is an example of an uncertainty measure in a Bayesian neural network to quantify epistemic uncertainty. Still, no analytic formula is known to describe it, one of the fundamental information measures to understand the Bayesian deep learning framework. In this paper, with the Dirichlet distribution assumption in its intermediate encoded message, we derive the analytical formula of the mutual information between model parameters and the predictive output by leveraging the notion of the point process entropy. Then, as an application, we discuss the estimation of the Dirichlet parameters and show its practical application in the active learning uncertainty measures.
翻译:Bayesian神经网络在许多应用问题上成功地设计和优化了强大的神经网络模型,包括不确定性量化。然而,由于最近取得了成功,关于Bayesian神经网络的信息理论理解仍处于早期阶段。相互信息是Bayesian神经网络中用于量化认知不确定性的不确定性措施的一个实例。然而,尚没有已知的解析公式来描述它,这是理解Bayesian深层学习框架的基本信息措施之一。在本文中,Dirichlet 分布假设在其中间编码电文中,我们通过利用点电动概念来获取模型参数和预测输出之间的相互信息分析公式。然后,作为应用,我们讨论Drichlet参数的估计,并展示其在积极学习不确定性措施中的实际应用情况。