Today, we move into the last module of the semester: Linear Algebra and Data! Data is all around us. Those who can analyze it have a leg up in almost every field. Data science is held up by the four pillars shown in this AI-generated image:

Machine learning is the subfield of data science concerned with the creation of algorithms that can be trained on data and can then classify or predict what will happen with new data. We've already seen the largest use of linear algebra in machine learning: least squares problems. These are at the heart of a large part of machine learning and artificial intelligence. In this module, we will see a few other key uses of linear algebra in data science.

A neural network is a mathematical attempt to mimic the amazing complexity of biological systems of neurons. Nothing we can create mathematically compares in complexity and scale to the human brain, but our artificial networks are still very complex! Here's a representation of a neural network:

^{Image from: https://www.ibm.com/think/topics/neural-networks}

Definition: the circles in this image are called nodes (or neurons), and the lines are called edges. Each node contains a single number, and each edge indicates which nodes will be used to calculate the numbers in subsequent nodes.

This giant, messy picture is really just representing a function (or, a composition of functions). Data goes in on the left (the input layer). From there, the number in each node of the subsequent layers is just a function of all the nodes from the previous layer. At the end, we have an output layer consisting of nodes, each containing a number.

So, yeah, this thing is a function. Data goes in on the left, and we have a bunch of numbers coming out on the right.

We will attempt to answer these questions using a familiar example, the MNIST digit handwriting recognition dataset.

Let's remember the scenario from Lab 1 in this class. We have images of handwritten digits from 0 to 9:

The images consist of 784 pixels, each of which is a grayscale number from 0 (black) to 1 (white). We put this information in a single $\large 784\times 1$ vector. We have a large set of these digits we can use to "train" a machine learning method, and then we hope the method will accurately identify any other handwritten digit we give it.

To say this another way, we hope to give a vector $\large \vec{v}\in\mathbb{R}^{784}$ as an input and get a vector $\large \vec{p}\in\mathbb{R}^{10}$ containing probabilities (such as: 10% chance this is a 4, 80% chance this is a 9, 2% chance it is a 6) as an output.

So we want to transform a vector in $\large \mathbb{R}^{784}$ to a vector in $\large \mathbb{R}^{10}$. This could be done with a single linear transformation matrix with size $\large 10\times 784$. There's no way this will work! Everyone's handwriting is different, so there can't be a single linear transformation to map an image to a number.

Here's an idea: instead of trying to go directly from an image to a digit, let's try to identify parts or features of digits and try to recognize those. For example, here we break down a few digits into features such as "line", "upper loop", and "lower loop":

Our intermediate layer could focus on these features. The function of the neural network could then be mapped out like this:

$$\large \begin{array}{lcccr} \text{Input} & &\text{Intermediate} & &\text{Output}\\ \begin{bmatrix} 0.98\\ 0.98\\ 0.96\\ 0.94\\ \vdots\\ 0.12\\ 0.07\\ 0.08\\ 0.03\\ \vdots\\ 0.99\\ 0.97\\ \end{bmatrix} & \xrightarrow{A} & \begin{bmatrix} 0.92 \text{ (upper loop) }\\ 0.04\\ 0.08\\ 0.86 \text{ (lower loop) }\\ 0.04 \\ \vdots\\ 0.01\\ \end{bmatrix} &\xrightarrow{B} & \begin{bmatrix} 0.12 (\text{ digit 0 })\\ 0.01 (\text{ digit 1 })\\ \vdots\\ 0.86 (\text{ digit 8 })\\ 0.03 (\text{ digit 9 })\\ \end{bmatrix}\\ \vec{v}\in\mathbb{R}^{784} & & \vec{z}\in\mathbb{R}^{128} & & \vec{p}\in\mathbb{R}^{10}\\ \end{array} $$

We see a transformation $\large A:\mathbb{R}^{784}\Rightarrow\mathbb{R}^{128}$ ($\large A$ is a $128\times 784$ matrix) that takes our input image and returns an intermediate vector focused on different features. Then a transformation $\large B:\mathbb{R}^{128}\Rightarrow\mathbb{R}^{10}$ ($\large B$ is a $10\times 128$ matrix) takes those features and computes an output vector containing probabilities for each digit. Our overall transformation can be represented as $\large T=BA$.

With a layer in between the input and output, we have some hope of dealing with some of the variety in handwriting. We can do even better if we add another intermediate layer! What if our structure is the following:

Input	Intermediate 1	Intermediate 2	Output
Digit	Little Features (curls, edges, corners)	Larger Features (loops, lines)	Output
$\large \vec{v}\in\mathbb{R}^{784}$	$\large \vec{z}_1\in\mathbb{R}^{128}$	$\large \vec{z}_2\in\mathbb{R}^{128}$	$\large \vec{p}\in\mathbb{R}^{10}$

The idea here is to use the first layer to identify many small features and to use those small features to identify larger identifying markers. An individual number in a node in Layer 2 is a linear combination of all the numbers in Layer 1.

We put this together and have our neural network.

We now have a structure for our neural network: $\large W_1$ is a $\large 128\times 784$ matrix, $\large W_2$ is a $\large 128 \times 128$ matrix, and $\large W_3$ is a $\large 10\times 128$ matrix. Overall, the network is a transformation $\large W=W_3W_2W_1$. Kind of...we hinted above at the fact that the transformations are actually nonlinear.

Here's what each transformation actually looks like:

$$\large \sigma\left(W\vec{v}+\vec{b}\right)$$

Two things here are nonlinear:

1. The function $\large \sigma$ is a nonlinear transformation of the form $\large \sigma = \frac{1}{1+e^{-x}}$. In addition to adding complexity, it also forces all node values to lie between 0 and 1, so we can interpret them as probabilities.

2. The vector $\large \vec{b}$ is called the "bias". Remember, a linear transformation must have the property that $\large T\left(\vec{0}\right)=\vec{0}$. Adding the bias means we no longer satisfy that property. The bias allows for some features to be considered more (or less) than others.

We have three transformations in our neural network:

• Layer 1: $\large \quad\vec{z}_1=\sigma\left(W_1\vec{v}+\vec{b}_1\right)$
• Layer 2: $\large \quad\vec{z}_2=\sigma\left(W_2\vec{z}_1+\vec{b}_2\right)$
• Layer 3: $\large \quad\vec{p}=\sigma\left(W_3\vec{z}_2+\vec{b}_3\right)$

Remember, the transformation $\large W_1\vec{v}$ is just a bunch of sums like the following:

$$\large W_{1,1}v_1+W_{1,2}v_2+\dots+W_{1,784}v_{784}$$

We call the numbers $\large W_{i,j}$ the weights. They are the coefficients on the nodes. We need to actually pick a bunch of numbers for weights and our bias vectors. How many numbers do we need to pick?

• In Layer 1, we have: $\large 784\times 128 (\text{ for }W_1) +128 (\text{ for the bias })=100,480$.
• In Layer 2, we have: $\large 128\times 128 + 128 =16,512$.
• In Layer 3, we have: $\large 128\times 10 + 10 =1,290$.

In total, there are 118,282 parameters for us to choose. (This would be considered a "small" neural network.) How do we choose all these numbers?

Making the best choice for our 118,282 parameters is an exercise in optimization. We have a large set of already known digits we use for training our network. We want to choose the parameters so that the network correctly classifies as much of the training set as possible.

There are many approaches to optimization:

• We could just pick parameters at random and then test the network. Keep the best set of random parameters. This is a poor approach!
• We could start at random and make small changes. If the small change results in a better network, we keep it. If not, we abandon it. This is known as a genetic algorithm, and it tends to slowly drift towards an optimal set of parameters.
• What we actually do is use Calculus! Remember from Calc III that the gradient always points in the steepest uphill direction. We use this fact to always move towards a better set of parameters. The optimization technique is known as gradient descent, and it is the machine we use to get our network in operating order.

If you are interested, check out these notes on the gradient descent optimization algorithm: Calc III Gradient Descent Notes. You will not be tested on this algorithm, but it's a cool idea that's worth checking out!

We presented a neural network with two intermediate layers, and we gave the layers an interpretation in terms of the problem (Layer 1 for small features, and Layer 2 for larger features). Think of this as more of a metaphor than a concrete description! In reality, we have over 100,000 parameters, and we optimize to make the best network possible.

We are throwing a huge number of degrees of freedom at a large pile of data, and we are letting the mathematics work out the best possible structure for the intermediate layers. These layers may end up with some recognizable structure, but they usually don't!

Why did we present the layers in terms of small/large features? Two reasons:

1. It's cope. As mathematicians, we are uncomfortable with things that are not fully explainable. Neural networks are big and messy, and we can't always explain why parameters and layers end up with the structure they have. There is a field of AI known as "interpretable AI" that focuses on making machine learning tractable, and we have experts in that field right here at Mines in Applied Mathematics and Statistics.
2. It's still a good metaphor. Our network layers may not correspond perfectly to the upper loop of an 8 or the lower corner of a 2, but this is a good way to think about the layers. Layer 1 combines all the input data in ways that end up being the most useful or successful. Layer 2 combines all the Layer 1 data in ways that end up being the most useful or successful.

We have here a relatively brief introduction to a large subject. We've encouraged the responsible and intelligent use of AI throughout this class in both HW and in the Labs. We feel it's important for people to know what happens under the hood with data science, artificial intelligence, machine learning, and large language models.

For hands-on experience, we will implement some aspects of this in an upcoming Lab.

You now know more than the vast majority of people about machine learning. If you are interested in knowing more, we recommend CSCI470: Intro to Machine Learning (and, if you like that class, there are follow up courses in Deep Learning and Advanced Machine Learning). This campus also offers various other courses in optimization, data science, statistics, and artificial intelligence.

I can:

• Recall definitions of nodes and edges in a neural network.
• Determine the nature of the linear transformations involved from an image of a neural network.
• Count the number of parameters involved in a neural network.