The Karhunen–Loève Expansion

In the context of Gaussian processes, when the covariance kernel is specified, it is possible to represent the process in terms of a basis of orthogonal functions, often given by certain types of orthogonal polynomials. This is part of the Karhunen–Loève theorem, which states that a zero-mean Gaussian process can be represented as an infinite linear combination of orthogonal functions, with the coefficients being uncorrelated random variables.

The precise form of the orthogonal functions depends on the covariance kernel of the Gaussian process. For certain choices of covariance kernel, these functions will be polynomials. For instance, when the covariance kernel is the covariance function of the standard Brownian motion, the orthogonal functions are the Hermite polynomials. For other choices of covariance kernel, the orthogonal functions might not be polynomials but could still be expressed in terms of certain special functions.

To determine if a function can be represented as a Gaussian process, the covariance kernel that generates the Gaussian process which has the function function as a realization needs to be identified. The identified covariance kernel then determines the corresponding orthogonal functions via the Karhunen–Loève expansion.

Once the orthogonal functions have been identified, the function (or indeed any realization of the Gaussian process) can be expressed as a linear combination of these functions. This expansion can make more apparent certain properties of the represented functions, including their zeros, in terms of the properties of the corresponding orthogonal functions and the coefficients in the expansion.

Here is a bit more detail.

Finding Orthogonal Basis for a Gaussian Process with Zero Mean and Given Covariance Function

Finding an orthogonal basis for a Gaussian process (GP) with a given covariance function and zero mean involves several mathematical steps, primarily in functional analysis.

Step 1: Covariance Function

Assume you have a covariance function $K(x, y)$ and a zero mean. The covariance function describes how correlated the outputs are for any two inputs $x$ and $y$.

Step 2: Eigenfunctions and Eigenvalues

The next step is to find the eigenfunctions $\phi_i(x)$ and eigenvalues $\lambda_i$ of the operator defined by the covariance function. This is usually done by solving the equation:

$$\int K(x, y) \phi_i(y) dy = \lambda_i \phi_i(x)$$

This problem is often referred to as a "Sturm-Liouville" problem.

Step 3: Orthogonal Basis

The eigenfunctions $\phi_i(x)$ will form an orthogonal basis under the inner product defined by the covariance function. These can be normalized to satisfy $\langle \phi_i, \phi_j \rangle = \delta_{ij}$, where $\delta$ is the Kronecker delta.

Step 4: GP Representation

Once you have the eigenfunctions and eigenvalues, the GP can be represented as:

$$f(x) = \sum_{i=0}^\infty \sqrt{\lambda_i} a_i \phi_i(x)$$

Here, $a_i$ are i.i.d standard normally distributed random variables.