formula list - feliyur/exercises GitHub Wiki

Derivatives

Function	$d/dx$
$x^T \Sigma^{-1} x = \lVert x \rVert^2_\Sigma$	$\nabla_x \left(\cdot\right) = 2\Sigma^{-1} x$
f(g(x))	$J_f\cdot J_g,~\nabla f^T \cdot J_g,~f^\prime \cdot \nabla g^T,f^\prime g^\prime,\nabla f^T g^\prime,~J_f\cdot\nabla g^T$
$\lVert f(x) \rVert^2_\Sigma$	$2f(x)^T\Sigma^{-1} J_f$

Trace

(From wikipedia).

Trace equals the sum of eigenvalues: $$tr(A) = \sum \lambda_i$$

Trace of cyclic permutations of matrix product (if defined) is equal: $$tr(ABC)=tr(CAB)=tr(BCA)$$ (If matrices are symmetric, then any permutation order is allowed).

Full characterization of trace: $$tr(A+B)=tr(A)+tr(B)$$ $$tr(cA)=c\cdot tr(A)$$ $$tr(AB)=tr(BA)$$

Neural Tangent Kernel (NTK)

$$k_0(x_i, x_j) = \operatorname{\mathbb{E}}\limits_{\theta_0\sim \mathbb{P}(\theta_0)} \left\langle \nabla_\theta f(x_i, \theta_0), \nabla_\theta f(x_j, \theta_0) \right\rangle$$