Uniform Convergence of Orthonormal Basis Projections in RKHS

Definition: Reproducing Kernel Hilbert Space

A Hilbert space $H$ of functions on a set $D$ is called a reproducing kernel Hilbert space (RKHS) if there exists a function $k: D \times D \to \mathbb{R}$ such that:

1. For every $x \in D$, the function $k_x(\cdot) = k(\cdot, x)$ belongs to $H$.
2. For every $x \in D$ and every $f \in H$, the reproducing property holds: $f(x) = \langle f, k_x \rangle_H$.

The function $k$ is called the reproducing kernel of $H$.

Definition: Orthonormal Basis in RKHS

A sequence of functions ${e_n}_{n=1}^{\infty} \subset H$ is an orthonormal basis of the RKHS $H$ if:

1. Orthonormality: For all indices $n, m$, $\langle e_n, e_m \rangle_H = \delta_{nm}$, where $\delta_{nm}$ is the Kronecker delta.
2. Completeness: The span of ${e_n}_{n=1}^{\infty}$ is dense in $H$, which means: a. For any $f \in H$, if $\langle f, e_n \rangle_H = 0$ for all $n$, then $f = 0$. b. Equivalently, every function $f \in H$ can be represented as

$$f = \sum_{n=1}^{\infty} \langle f, e_n \rangle_H e_n$$

with convergence in the $H$-norm:

$$\lim_{N\to\infty} \left|f - \sum_{n=1}^{N} \langle f, e_n \rangle_H e_n\right|_H = 0$$

3. Parseval's Identity: For any $f \in H$,

$$|f|H^2 = \sum{n=1}^{\infty} |\langle f, e_n \rangle_H|^2$$

In an RKHS, each basis function satisfies the reproducing property: $e_n(x) = \langle e_n, k(\cdot,x) \rangle_H$ for all $x \in D$.

Theorem: Uniform Convergence

Let $H$ be a reproducing kernel Hilbert space (RKHS) on a set $D$ with reproducing kernel $k$. Suppose that:

1. ${e_n}_{n=1}^{\infty}$ is an orthonormal basis of $H$ as defined above.
2. The kernel is uniformly bounded on $D$; that is, there exists a constant $M > 0$ such that

$$\sup_{x \in D} |k(\cdot, x)|_H \le M$$

Then for any function $f \in H$ with orthonormal expansion

$$f = \sum_{n=1}^{\infty} c_n e_n$$

where $c_n = \langle f, e_n \rangle_H$, the partial sums

$$S_N f = \sum_{n=1}^N c_n e_n$$

converge uniformly to $f$ on $D$; in other words,

$$\lim_{N\to\infty} \sup_{x\in D} \bigl| S_N f(x) - f(x) \bigr| = 0$$

Proof

By the completeness property of the orthonormal basis, every function $f \in H$ can be represented by its orthonormal expansion that converges in the $H$-norm. Since $H$ is an RKHS, the evaluation functional at any $x \in D$ satisfies:

$$\bigl| f(x) - S_N f(x) \bigr| = \Bigl| \langle f - S_N f, k(\cdot, x) \rangle_H \Bigr|$$

Using the Cauchy-Schwarz inequality:

$$\bigl| f(x) - S_N f(x) \bigr| \le |f - S_N f|_H , |k(\cdot, x)|_H$$

Taking the supremum over $x \in D$ yields:

$$\sup_{x \in D} \bigl| f(x) - S_N f(x) \bigr| \le |f - S_N f|H , \sup{x \in D} |k(\cdot, x)|_H$$

By the uniform boundedness assumption:

$$\sup_{x \in D} \bigl| f(x) - S_N f(x) \bigr| \le M , |f - S_N f|_H$$

From the convergence property of orthonormal bases:

$$\lim_{N\to\infty} |f - S_N f|_H = 0$$

For any $\varepsilon > 0$, choose $N$ such that for all $n \ge N$:

$$|f - S_n f|_H < \frac{\varepsilon}{M}$$

Then for all $n \ge N$:

$$\sup_{x \in D} \bigl| f(x) - S_n f(x) \bigr| < \varepsilon$$

Thus:

$$\lim_{N\to\infty} \sup_{x\in D} |S_N f(x) - f(x)| = 0$$

Remarks

1. The uniform boundedness condition on $|k(\cdot, x)|_H$ is essential. Without it, norm convergence in the RKHS does not guarantee uniform convergence of evaluations on $D$. This condition ensures the kernel's feature maps are uniformly bounded across the entire domain.

2. The domain $D$ need not be compact. The result holds for arbitrary domains (e.g., $D = \mathbb{R}^n$) as long as $\sup_{x \in D} |k(\cdot, x)|_H \le M$.

3. The uniform convergence applies to expansions of functions $f \in H$ in any orthonormal basis. For expansions of the kernel $k(x, y)$ itself, uniform convergence holds only for the Mercer eigenbasis ${e_n^*}$ satisfying:

$$\int_D k(x, y)e_n^(y),dy = \lambda_n e_n^(x)$$

Non-Mercer bases yield pointwise convergence.

Bibliography

1. Riesz, F. (1907). Sur les systèmes orthogonaux de fonctions. Comptes rendus de l'Académie des sciences, 144:615--619.

2. Fischer, E. (1907). Sur la convergence en moyenne. Comptes rendus de l'Académie des sciences, 144:1022--1024.

3. Aronszajn, N. (1950). Theory of reproducing kernels. Transactions of the American Mathematical Society, 68(3):337--404.

4. Berlinet, A. and Thomas-Agnan, C. (2004). Reproducing Kernel Hilbert Spaces in Probability and Statistics. Springer, Boston, MA.