Post-approval safety surveillance of medical products using observational healthcare data can help identify safety issues beyond those found in pre-approval trials. When testing sequentially as data accrue, maximum sequential probability ratio testing (MaxSPRT) is a common approach to maintaining nominal type 1 error. However, the true type 1 error may still deviate from the specified one because of systematic error due to the observational nature of the analysis. This systematic error may persist even after controlling for known confounders. Here we propose to address this issue by combing MaxSPRT with empirical calibration. In empirical calibration, we assume uncertainty about the systematic error in our analysis, the source of uncertainty commonly overlooked in practice. We infer a probability distribution of systematic error by relying on a large set of negative controls: exposure-outcome where no causal effect is believed to exist. Integrating this distribution into our test statistics has previously been shown to restore type 1 error to nominal. Here we show how we can calibrate the critical value central to MaxSPRT. We evaluate this novel approach using simulations and real electronic health records, using H1N1 vaccinations during the 2009-2010 season as an example. Results show that combining empirical calibration with MaxSPRT restores nominal type 1 error. In our real-world example, adjusting for systematic error using empirical calibration has a larger impact than, and hence is just as essential as, adjusting for sequential testing using MaxSPRT. We recommend performing both, using the method described here.
翻译:使用观察保健数据对医疗产品进行事后批准后的安全监控,可以帮助确定在批准前试验中发现的安全问题之外的其他安全问题。在数据累积时,按顺序进行测试时,最大序列概率比测试(MaxSPRT)是维持名义型1错误的一种常见方法。然而,由于分析的观察性质,真正的第1型错误仍可能与特定错误不同,因为有系统错误,分析的观察性质导致的系统错误。即使对已知的混杂者进行了控制,这种系统错误也可能继续存在。我们在这里建议用经验校准来梳理MaxSPRT。在经验校准中,我们假设我们的分析系统错误存在不确定性,不确定性的来源在实践中经常被忽视。我们推论的是,依靠大量负面控制(MaxSPRT)来判断系统错误的概率分布:在认为不存在因果关系的暴露结果的情况下,将这种分布纳入我们的测试统计系统化的系统错误恢复到名义上。在这里我们展示了如何将关键值核心值与MaxSPRT调校准到这里。我们用模拟和真实的电子健康记录来评估这一新方法,在2009-2010年季节使用H1N1疫苗进行模拟和真实的接种,作为例子。我们使用系统化的校正校准方法来进行一次的校准,因此,使用一个比重的校准方法进行一次校准。