Common statistical measures of uncertainty such as $p$-values and confidence intervals quantify the uncertainty due to sampling, that is, the uncertainty due to not observing the full population. However, sampling is not the only source of uncertainty. In practice, distributions change between locations and across time. This makes it difficult to gather knowledge that transfers across data sets. We propose a measure of uncertainty or instability that quantifies the distributional instability of a statistical parameter with respect to Kullback-Leibler divergence, that is, the sensitivity of the parameter under general distributional perturbations within a Kullback-Leibler divergence ball. In addition, we propose measures to elucidate the instability of parameters with respect to directional or variable-specific shifts. Measuring instability with respect to directional shifts can be used to detect the type of shifts a parameter is sensitive to. We discuss how such knowledge can inform data collection for improved estimation of statistical parameters under shifted distributions. We evaluate the performance of the proposed measure on real data and show that it can elucidate the distributional (in-)stability of a parameter with respect to certain shifts and can be used to improve the accuracy of estimation under shifted distributions.
翻译:通常的不确定性统计计量方法,如美元价值和信心间隔,以数量表示抽样造成的不确定性,即不观察整个人口造成的不确定性。不过,抽样并不是唯一的不确定性来源。在实践中,地点之间和时间之间的分布变化。这使得难以收集跨数据集传输的知识。我们建议一种不确定性或不稳定度,以数量表示与Kullback-Lebeller差异有关的统计参数分布不稳定性,即Kullback-Lebeller差异球内一般分布扰动下参数的敏感度。此外,我们建议采取措施,说明关于方向性或具体变异性变化的参数的不稳定性。测量方向性变化的不稳定性,可以用来检测一个参数的转移类型。我们讨论这种知识如何为数据收集提供信息,以改进对转移分布下统计参数的估计。我们评估了拟议数据计量的性能,并表明该参数与某些变化有关的分布(内在)可靠性。我们还可以使用这种知识来说明某一参数的分布情况的准确性。