The final piece in our theoretical module deals with orthogonality. Orthogonal vectors are our preferred vectors for most uses, so we need to look more closely at them. Everything in here is about orthogonality and how we can get it or use it. It's not just a random list of facts about orthogonality, though. The ideas hopefully build on each other in a logical way.

As a reminder, our key fact about orthogonality is:

Two vectors are orthogonal if their dot product is 0.

We'll start with a definition:

Definition: Let $\large W$ be a subspace of $\large \mathbb{R}^n$. The set of all vectors orthogonal to $\large W$ is called the orthogonal complement of $\large W$, and we write $\large W^\perp$.

As an example, let $\large W$ be a plane through the origin in $\large \mathbb{R}^3$ (we've seen that a plane is a subspace of $\large \mathbb{R}^3$). What is $\large W^\perp$? What vectors are orthogonal to the plane? The image reminds us that there is a single direction perpendicular to a plane:

There is a single basis vector orthogonal to the plane (the blue vector). The subspace is the set of all linear combinations of that basis vector. In other words, $\large W^\perp$ is a line in $\large\mathbb{R}^3$ (the red line).

Here's a neat result with orthogonal subspaces. Suppose we have a subspace $\large W$ of $\large \mathbb{R}^n$. Every vector $\large \vec{x}\in \mathbb{R}^n$ is the sum of a vector $\large \hat{x}\in W$ and a vector $\large \vec{z}\in W^\perp$. That is $\large \vec{x}=\hat{x}+\vec{z}$.

There is a theorem that we will use in optimization problems in our next module:

Theorem: Let $\large A$ be an $\large m\times n$ matrix. Then $\large \left( \text{Row}(A)\right)^\perp=\text{Null}(A)$ and $\large \left( \text{Col}(A)\right)^\perp=\text{Null}(A^T)$.

At first glance, this looks like the kind of thing you'd need to think hard about to understand. It's not that bad though. For the first statement, let's remember that $\large \text{Null}(A)$ is made up of solutions of $\large A\vec{x}=\vec{0}$, and $\large A\vec{x}$ is just a bunch of dot products between the rows of $\large A$ and $\large \vec{x}$. So, yeah, the rows (and linear combinations of the rows) of $\large A$ are orthogonal to anything in $\large \text{Null}(A)$.

The second statement is just the first statement applied to the matrix $\large A^T$ (since the rows of $\large A$ are the columns of $\large A^T$).

For reasons we will see throughout the rest of the class, orthogonal vectors are the best vectors to work with!

Definition: A set of vectors $\large \vec{v_1},\vec{v_2},\dots,\vec{v_m}$ in $\large \mathbb{R}^n$ is an orthogonal set of vectors if each pair of vectors is orthogonal (if $\large \vec{v_i}\cdot\vec{v_j}=0$ for all $\large i\ne j$).

Example: Show that the following is an orthogonal set of vectors:

$$\large \vec{v_1}=\begin{bmatrix} 3 \\ 1\\ 1\\ \end{bmatrix}, \vec{v_2}=\begin{bmatrix} -1 \\ 2\\ 1\\ \end{bmatrix}, \vec{v_3}=\begin{bmatrix} -1/2 \\ -2\\ 7/2\\ \end{bmatrix} $$

We'll see in the video that these do make up an orthogonal set.

Now, a theorem that illustrates why we care about orthogonal sets of vectors:

Theorem: An orthogonal set of nonzero vectors is linearly independent.

The proof is short and is a good example of how these types of arguments are made. Please watch it in the video:

Definition: An orthogonal basis for a subspace is a basis that is also an orthogonal set.

Now, in general if we want to write a given vector as a linear combination of basis vectors, we need to solve a system of linear equations (Gaussian elimination...yay!) to find the coefficients in the linear combination. But if we have an orthogonal basis, it's much easier!

Theorem: If $\large \lbrace \vec{v_1},\vec{v_2},\dots,\vec{v_m}\rbrace$ is an orthogonal basis for $\large W$, then any $\large \vec{u}\in W$ can be written $\large \vec{u} = c_1\vec{v_1}+c_2\vec{v_2}+\dots+c_m\vec{v_m}$ where the coefficients are:

$$\large c_j = \frac{\vec{u}\cdot\vec{v_j}}{\vec{v_j}\cdot\vec{v_j}}$$

Here's an example of how to use the theorem. We found earlier that the set:

$$\large \vec{v_1}=\begin{bmatrix} 3 \\ 1\\ 1\\ \end{bmatrix}, \vec{v_2}=\begin{bmatrix} -1 \\ 2\\ 1\\ \end{bmatrix}, \vec{v_3}=\begin{bmatrix} -1/2 \\ -2\\ 7/2\\ \end{bmatrix} $$

is an orthogonal set in $\large \mathbb{R}^3$. Since it's orthogonal, it's linearly independent. A linearly independent set of 3 vectors in $\large \mathbb{R}^3$ is necessarily a basis for $\large \mathbb{R}^3$. So we have an orthogonal basis. Write the vector $\large \left<6,1,-8\right>$ as a linear combination of these basis vectors. The solution is in the video:

We can repeat all of the above with unit vectors (vectors of length one, $\large \lVert\vec{v}\rVert=1$), and we get definitions for orthonormal set and orthonornal basis. If we have an orthonormal basis the formula for coefficients in a linear combination of basis vectors is even simpler. Since $\large \vec{v_j}\cdot\vec{v_j}=\lVert\vec{v_j}\rVert^2=1$, we get

$$\large c_j = \vec{u}\cdot\vec{v_j}$$

In Calc II, we learned about the projection of one vector onto another. The projection of $\large \vec{u}$ onto $\large \vec{v}$ is a measure of how much of $\large \vec{u}$ points in the same direction as $\large \vec{v}$. It breaks $\large \vec{u}$ into a vector parallel to $\large \vec{v}$ and a vector orthogonal to $\large \vec{v}$. In the pictures, we see examples of projections of vectors $\large \vec{u}$ (green) onto vectors $\large \vec{v}$ (blue). The red vectors are the projections, the components of $\large \vec{u}$ in the direction of $\large \vec{v}$. The dotted black vectors are the components of $\large \vec{u}$ orthogonal to $\large \vec{v}$.

The formula we learned in calculus for the projection of $\large \vec{u}$ onto $\large \vec{v}$ is:

$$\large \text{proj}_{\vec{v}}\vec{u}=\frac{\vec{u}\cdot\vec{v}}{\lVert \vec{v}\rVert ^2}\vec{v}$$

Even though we can only visualize projections in $\large \mathbb{R}^2$ and $\large \mathbb{R}^3$, we can still calculate them in $\large \mathbb{R}^n$. The formula is the same! Again, this will break up a vector into components that are parallel and orthogonal to a given direction.

Now, here's a new idea, and a big one. Instead of the projection of $\large \vec{u}$ onto a vector, we will define the orthogonal projection of $\large \vec{u}$ onto an entire subspace! We can visualize this in $\large \mathbb{R}^3$. Let $\large W$ be a plane (which, as we've seen, is a subspace). The projection of $\large \vec{u}$ onto this subspace is shown (vector is blue, projection onto $\large W$ is red, orthogonal component is purple):

The idea of the projection is that any vector $\large \vec{u}\in \mathbb{R}^n$ can be split up into two orthogonal components, one that is in the subspace $\large W$ and one that is in $\large W^\perp$. That is, $\large \vec{u}=\hat{u}+\vec{z}$, where $\large \hat{u}\in W$ and $\large \vec{z}\in W^\perp$. We call $\large \hat{u}$ the orthogonal projection of $\large \vec{u}$ onto $\large W$, and we write $\large \hat{u}=\text{proj}_W\vec{u}$.

Theorem: Let $\large \hat{u}$ be the projection of $\large \vec{u}$ onto a subspace $\large W$, $\large \hat{u}=\text{proj}_W\vec{u}$. Then $\large\lVert \vec{u}-\hat{u}\rVert \le \lVert \vec{u}-\vec{v}\rVert$ for any $\large \vec{v}\in W$.

We now have a picture and an idea. We'd like to project a vector onto a subspace; we'd like to find the vector in $\large W$ closest to $\large \vec{u}$. How do we do this? The idea is to strip away, one at a time, every part of $\large \vec{u}$ that is orthogonal to the vectors in $\large W$. If we can do that, we will be left with a vector $\large \hat{u}=\text{proj}_W\vec{u}$ that is parallel to everything in $\large W$.

Here's our situation and formulas, which are based on the idea of subtracting off each orthogonal direction from $\large \vec{u}$:

Theorem: Let $\large \vec{u}$ be a vector in $\large \mathbb{R}^n$, and let $\large W$ be a subspace of $\large \mathbb{R}^n$ with orthogonal basis $\large \lbrace \vec{v_1}, \vec{v_2},\dots,\vec{v_m} \rbrace$. Then there are vectors $\large \hat{u}\in W$ and $\large \vec{z}\in W^\perp$ such that $\large \vec{u}=\hat{u}+\vec{z}$. The formulas for $\large \hat{u}$ and $\large \vec{z}$ are:

$$\large \begin{split} \vec{z}&=\vec{u}-\frac{\vec{u}\cdot\vec{v_1}}{\lVert \vec{v_1} \rVert ^2} \vec{v_1} -\frac{\vec{u}\cdot\vec{v_2}}{\lVert \vec{v_2} \rVert ^2} \vec{v_2} -\dots -\frac{\vec{u}\cdot\vec{v_m}}{\lVert \vec{v_m} \rVert ^2} \vec{v_m}\\ \hat{u}&=\text{proj}_W\vec{u}=\frac{\vec{u}\cdot\vec{v_1}}{\lVert \vec{v_1} \rVert ^2} \vec{v_1} +\frac{\vec{u}\cdot\vec{v_2}}{\lVert \vec{v_2} \rVert ^2} \vec{v_2} +\dots +\frac{\vec{u}\cdot\vec{v_m}}{\lVert \vec{v_m} \rVert ^2} \vec{v_m}\\ \end{split} $$

This is a lot, and it's and the end of one of our longest lectures all semester. Let's do an example that helpfully makes it more clear. The vectors $\large \vec{v_1}=\left< 2,5,-1\right>$ and $\large \vec{v_2}=\left<-2, 1, 1\right>$ form an orthogonal (test them!) basis for $\large W=\text{Span}\lbrace \vec{v_1},\vec{v_2}\rbrace$, a plane in $\large \mathbb{R}^3$. Find the orthogonal projection of $\large \vec{u}=\left<1,2,3\right>$ onto this plane. The work can be seen in the video:

The image below shows the plane and vectors in this particular example:

I can:

• Recall and use the definitions of orthogonal set, orthogonal basis, orthogonal subspace, and orthogonal projection.
• Recognize an othogonal/orthonormal set of vectors.
• Recall and use theorems related to orthogonal/orthonormal sets/bases.
• Write a vector as a linear combination of an orthogonal/orthonormal basis.
• Find the orthogonal projection of a vector onto a subspace.