Model-Free Error Assessment for Breadth-First Studies, with Applications to Cell-Perturbation Experiments

We consider high-throughput experiments that take measurements regarding many parameters. Due to resource limitations, ``breadth-first'' high-throughput experiments take only a few independent samples of each parameter, and so it is challenging to assess estimator error. We propose a new model-free method for bounding type S errors in this context, based on a quantity we call the Cross-replicate Sign Error Rate (CSER). The CSER is the expected sign agreement between a fixed set of estimates and estimates based on an independent experimental replicate. To show the CSER can be estimated with enough accuracy to be useful in practice, we develop new improvements to Hoeffding's bounds for sums of bounded random variables, obtaining the tightest bounds that can be obtained from the Chernoff inequality. We apply this method to analyzing measurements from cell-perturbation experiments. Our method reveals that existing error control practices fail to control error at their nominal level in some cases and are needlessly conservative in others. The CSER is easy to estimate, enabling practitioners to detect problems in their experimental designs and identify subsets of parameters with a low proportion of type S errors.