Covariate Balancing Methods for Randomized Controlled Trials Are Not Adversarially Robust

The first step towards investigating the effectiveness of a treatment is to split the population into the control and the treatment groups, then compare the average responses of the two groups to the treatment. In order to ensure that the difference in the two groups is only caused by the treatment, it is crucial for the control and the treatment groups to have similar statistics. The validity and reliability of trials are determined by the similarity of two groups' statistics. Covariate balancing methods increase the similarity between the distributions of the two groups' covariates. However, often in practice, there are not enough samples to accurately estimate the groups' covariate distributions. In this paper, we empirically show that covariate balancing with the standardized means difference covariate balancing measure is susceptible to adversarial treatment assignments in limited population sizes. Adversarial treatment assignments are those admitted by the covariate balance measure, but result in large ATE estimation errors. To support this argument, we provide an optimization-based algorithm, namely Adversarial Treatment ASsignment in TREatment Effect Trials (ATASTREET), to find the adversarial treatment assignments for the IHDP-1000 dataset.