Locally Optimal Design for A/B Testing in the Presence of Covariates and Network Connection

A/B testing, also known as controlled experiments, refers to the statistical procedure of conducting an experiment to compare two treatments applied to different testing subjects. For example, many IT companies frequently conduct A/B testing experiments on their users who are connected and form social networks. Often, the users' responses could be related to the network connection. In this paper, we assume that the users, or the test subjects of the experiments, are connected on an undirected network, and the responses of two connected users are correlated. We include the treatment assignment, covariates features, and network connection in a conditional autoregressive model. Based on this model, we propose a design criterion that measures the variance of the estimated treatment effect and allocate the treatment settings to the test subjects by minimizing the criterion. Since the design criterion depends on an unknown network correlation parameter, we adopt the locally optimal design method and develop a hybrid optimization approach to obtain the optimal design. Through synthetic and real social network examples, we demonstrate the value of including network dependence in designing A/B testing experiments and validate that the proposed locally optimal design is robust to the choices of parameters.