Conditional and transductive inference about populations with fixed attributes

It has been argued that parameters that characterize sub-populations can be more relevant than super-population parameters. For example, a video subscription service might be interested in estimating the satisfaction of its current customers, as opposed to estimating that of a hypothetical infinite super-population. In this case, the customers might be viewed as fixed, while the satisfaction measurements might be random due to measurement noise and temporal variation. More generally, inference for populations with fixed attributes can be modeled as inferring parameters of conditional distributions given these attributes. Since the data for such sub-population are drawn from a conditional distribution, it is desirable that confidence intervals have conditional coverage guarantees, as opposed to marginal coverage guarantees. We provide a framework for statistical inference on parameters of sub-populations with fixed attributes. We construct confidence intervals that attain asymptotic validity given the attributes. In addition, we develop a set of tools to infer the parameters of new populations with observed attributes under covariate shift; the confidence intervals also attain asymptotic conditional validity under mild conditions. The validity and applicability of the proposed methods are demonstrated on simulated and real-world data.