In linear regression we wish to estimate the optimum linear least squares predictor for a distribution over $d$-dimensional input points and real-valued responses, based on a small sample. Under standard random design analysis, where the sample is drawn i.i.d. from the input distribution, the least squares solution for that sample can be viewed as the natural estimator of the optimum. Unfortunately, this estimator almost always incurs an undesirable bias coming from the randomness of the input points, which is a significant bottleneck in model averaging. In this paper we show that it is possible to draw a non-i.i.d. sample of input points such that, regardless of the response model, the least squares solution is an unbiased estimator of the optimum. Moreover, this sample can be produced efficiently by augmenting a previously drawn i.i.d. sample with an additional set of $d$ points, drawn jointly according to a certain determinantal point process constructed from the input distribution rescaled by the squared volume spanned by the points. Motivated by this, we develop a theoretical framework for studying volume-rescaled sampling, and in the process prove a number of new matrix expectation identities. We use them to show that for any input distribution and $\epsilon>0$ there is a random design consisting of $O(d\log d+ d/\epsilon)$ points from which an unbiased estimator can be constructed whose expected square loss over the entire distribution is bounded by $1+\epsilon$ times the loss of the optimum. We provide efficient algorithms for generating such unbiased estimators in a number of practical settings and support our claims experimentally.
翻译:在线性回归中,我们希望根据一个小样本来估计用于美元维度输入点和真实价值响应的分布的最佳线性最小正方值预测值。 在标准随机设计分析中,如果从输入分布中抽取样本,那么该样本的最小正方数解决方案可以被视为最佳的自然估计值。 不幸的是,这个估计值几乎总是从输入点的随机性中得出一个不可取的偏差,这是模型平均比例中一个很大的瓶颈。在本文中,我们显示有可能绘制一个非i.i.d. 的输入点样本,这样,不管响应模型,最小方数的最小正方位解决方案是最佳分布的公正估计值。此外,这一样本可以通过增加一个先前绘制的i.i.i.d.d.样本,加上一组美元点的附加的确定点,它由输入量重新标定的输入量构成一个一定的确定点(我们平面数量,我们为此开发了一个理论框架框架,用来研究正方位值的正方位值值和正方位数的数值值值值值计算出一个预估测算的数值,在任何正方位值的计算中,从而显示我们正方位的正方位计算值的计算值的计算结果的计算结果的计算值的计算结果的计算结果中,在任何正方位值的正方位值的正方位值运价值运价值的计算结果中,在任何正方位值的计算值的计算值的计算结果中,在任何正方价值运价值运价值运价值的计算值的计算值计算值的计算出一个正方值的计算出一个正方位值运价值运价值运价值的计算值的计算值的数值值的数值值的计算值的数值值的数值。。