We study the consistency and optimality of the maximum marginal likelihood estimate (MMLE) in the hyperparameter inference for large-degree-of-freedom models. We perform main analyses within the exponential family, where the natural parameters are hyperparameters. First, we prove the consistency of the MMLE for the general linear models when estimating the scales of variance in the likelihood and prior. The proof is independent of the number ratio of data to model parameters and excepts the ill-posedness of the associated regularized least-square model-parameter estimate that is shown asymptotically unbiased. Second, we generalize the proof to other models with a finite number of hyperparameters. We find that the extensive properties of cost functions in the exponential family generally yield the consistency of the MMLE for the likelihood hyperparameters. Besides, we show the MMLE asymptotically almost surely minimizes the Kullback-Leibler divergence between the prior and true predictive distributions even if the true data distribution is outside the model space under the hypothetical asymptotic normality of the predictive distributions applicable to non-exponential model families. Our proof validates the empirical Bayes method using the hyperparameter MMLE in the asymptotics of many model parameters, ensuring the same qualification for the empirical-cross-entropy cross-validation.
翻译:我们研究了高度自由模型超参数推断值中最大边际概率估计值(MMLE)的一致性和最佳性。我们研究了大度自由模型超度模型超度参数的最大边际概率估计值(MMLE)的一致性和最佳性。我们研究了超度参数参数的超度模型内的主要分析。首先,在估计概率和先前差异尺度时,我们证明了普通线性模型的MMLE值对一般线性模型的一致性。此外,我们从数据与模型参数的数值比重上看,证据几乎可以肯定地将数据与模型参数之间相关最正常最低方位模型参数估计值之间的误差降到最低。第二,我们用一定数量的超度参数将证据推广到其他模型中的其它模型。我们发现,指数性模型中成本功能的广泛特性一般能产生MMLE值对超度参数的一致性。此外,我们从整体上看,MMLE几乎肯定地将数据先前和真实预测性最低比值分布值之间的误差降到最低,即使真实性模型在假设的范围之外,但真正的数据分布是有限的。我们预测性标准模型的正常性参数分布是用于非实验性模型的,我们用于用于非实验性模型。