Scale invariant process regression

Gaussian processes are the leading method for non-parametric regression on small to medium datasets. One main challenge is the choice of kernel and optimization of hyperparameters. We propose a novel regression method that does not require specification of a kernel, length scale, variance, nor prior mean. Its only hyperparameter is the assumed regularity (degree of differentiability) of the true function. We achieve this with a novel non-Gaussian stochastic process that we construct from minimal assumptions of translation and scale invariance. The process can be thought of as a hierarchical Gaussian process model, where the hyperparameters have been incorporated into the process itself. To perform inference with this process we develop the required mathematical tools. It turns out that for interpolation, the posterior is a t-process with a polyharmonic spline as mean. For regression, we state the exact posterior and find its mean (again a polyharmonic spline) and approximate variance with a sampling method. Experiments show a performance equal to that of Gaussian processes with optimized hyperparameters. The most important insight is that it is possible to derive a working machine learning method by assuming nothing but regularity and scale- and translation invariance, without any other model assumptions.