Regularization of the ensemble Kalman filter using a non-parametric, non-stationary spatial model

The sample covariance matrix of a random vector is a good estimate of the true covariance matrix if the sample size is much larger than the length of the vector. In high-dimensional problems, this condition is never met. As a result, in high dimensions the EnKF ensemble does not contain enough information to specify the prior covariance matrix accurately. This necessitates the need for regularization of the analysis (observation update) problem. We propose a regularization technique based on a new spatial model on the sphere. The model is a constrained version of the general Gaussian process convolution model. The constraints on the location-dependent convolution kernel include local isotropy, positive definiteness as a function of distance, and smoothness as a function of location. The model allows for a rigorous definition of the local spectrum, which, in addition, is required to be a smooth function of spatial wavenumber. We regularize the ensemble Kalman filter by postulating that its prior covariances obey this model. The model is estimated online in a two-stage procedure. First, ensemble perturbations are bandpass filtered in several wavenumber bands to extract aggregated local spatial spectra. Second, a neural network recovers the local spectra from sample variances of the filtered fields. We show that with the growing ensemble size, the estimator is capable of extracting increasingly detailed spatially non-stationary structures. In simulation experiments, the new technique led to substantially better EnKF performance than several existing techniques.