Adaptive design optimization (ADO) is a state-of-the-art technique for experimental design (Cavagnaro, Myung, Pitt, & Kujala, 2010). ADO dynamically identifies stimuli that, in expectation, yield the most information about a hypothetical construct of interest (e.g., parameters of a cognitive model). To calculate this expectation, ADO leverages the modeler's existing knowledge, specified in the form of a prior distribution. Informative priors align with the distribution of the focal construct in the participant population. This alignment is assumed by ADO's internal assessment of expected information gain. If the prior is instead misinformative, i.e., does not align with the participant population, ADO's estimates of expected information gain could be inaccurate. In many cases, the true distribution that characterizes the participant population is unknown, and experimenters rely on heuristics in their choice of prior and without an understanding of how this choice affects ADO's behavior. Our work introduces a mathematical framework that facilitates investigation of the consequences of the choice of prior distribution on the efficiency of experiments designed using ADO. Through theoretical and empirical results, we show that, in the context of prior misinformation, measures of expected information gain are distinct from the correctness of the corresponding inference. Through a series of simulation experiments, we show that, in the case of parameter estimation, ADO nevertheless outperforms other design methods. Conversely, in the case of model selection, misinformative priors can lead inference to favor the wrong model, and rather than mitigating this pitfall, ADO exacerbates it.
翻译:自适应设计优化(ADO)是实验设计的最先进技术之一。ADO动态识别那些期望最有信息地关于感兴趣的假设构造上数据(例如,Cognitive model的参数)的刺激。为了计算这个期望,ADO利用建模者已有的知识,以先验分布的形式给出。信息量丰富的先验分布与参与者人群中的要素分布保持一致。ADO内部评估期望的信息增益就是按照这个先验分布进行的。如果先验不准确,即与参与者人群不一致,ADO预期的信息增益估计就可能出现误差。在许多情况下,表征参与者人群的真实分布是未知的,实验者在选择先验时就要借助经验法则,而没有理解这个选择如何影响ADO的行为。我们的工作介绍了一个数学框架,有助于探究先验分布选择对使用ADO设计实验的效率造成的影响。通过理论和实证结果,我们表明,在先验不准确的情况下,关于期望信息增益的度量与相应推断的正确性是不同的。通过一系列模拟实验,我们表明,在参数估计方面,ADO仍然优于其他设计方法。相反,在模型选择方面,不正确的先验分布可能导致推断偏向错误的模型,ADO则加剧了这个问题。