Stats 101 - amitkhopkar-creator/GNN_Architecture GitHub Wiki

1. Central Tendency

There are several measures of central tendency such as Mean, Median and Mode for a given data. We can derive the central tendencies of an example dataset D = {3,4,5,7,8,9,6,3,5,4,4,1,2,2}

Mean

Mean is calculated by adding all the elements in the the dataset and dividing by the count of numbers. For a population mean it is depicted by the letter (u) and for a sample mean its (x-bar).

Median

arrange numbers in the dataset in an ascending list and pick the number in the centre

Mode

Most repeated. There is always one mean and one median but there may be multiple modes.

Variance

Variance measures how one number in the dataset spread out from its own mean. A measurement of how far each number in a data set is from the mean and thus from every other number in the set. Variance is often depicted by this symbol: σ2 It is calculated by taking the differences between each number in the data set and the mean, squaring the differences to make them positive, and then dividing the sum of the squares by the number of values in the population. Divide by n-1 for a sample.

Standard Deviation

It is the square root of variance

Covariance

For e.g. if we need a model to predict exam scores. we can plot hours of study (x) vs exam score (y) for a group of students.

Positive covariance

students who study more than the average hours tend to score higher than the average score

Negative Covariance

we could take another example where hours spent playing video games (x) vs exam score (y). A higher playing time may lead to a lower score than average.

Zero Covariance

e.g. Students shoe size (x) vs exam scores (y)

Calculating Covariance

Covariance (x,y) = E [(Xi - X-bar) x (yi - y-bar)] / n-1

If a datapoint (x,y) is such that x and y are both greater than x-bar and y-bar respectively then we multiply two positive numbers which leads to a positive result. Similarly if x and y are lower than x-bar and y-bar w multiply two negative numbers which leads to a positive result.

Adding up all these numbers gives a positive overall covariance.

If a datapoint (x,y) is such that x might be lower than x-bar and y might be higher than y-bar, then the multiplication yields a negative number. Similarly when x is greater than x-bar and y is lower than y-bar, then again the multiplication yields a negative number.

The sum of all such negative numbers gives us a negative Covariance.

Covariance signals the direction (positive or negative) but the number could be arbitrary. For e.g. if height in meters Vs weight in Kg is measured, we get one covariance output. If instead the heights is in cms, the covariance output changes significantly.

Therefore covariance confirms directionality but the number if not much useful.

Correlation (Standardising the Relationship)

To fix the unit problem, statisticians created Correlation (specifically, Pearson's Correlation Coefficient, written as r).

Correlation takes the covariance and divides it by the product of standard deviations of both variables.

Because standard deviation also uses the same units as the data, dividing by them completely strips the units away. It leaves us with a pure, standardized number that always falls between -1 and +1. How to Interpret the Number (r):

• (r = +1): A perfect positive linear relationship. If you plot it, all data points sit perfectly on a straight upward line.

• (r = -1): A perfect negative linear relationship. All points sit perfectly on a straight downward line.

• (r = 0): No linear relationship at all. The points look like a random cloud.

• Value strength: The closer the number is to (+1) or (-1) (e.g., (0.85) or (-0.72)), the stronger and tighter the relationship is. The closer it is to (0) (e.g., (0.15)), the weaker it is.

Going back to our height and weight example: whether you measure height in metres, centimetres, or inches, the correlation (r) will remain exactly the same.

In the (MSE) mean squared error formula why do we multiply by 2n instead on n the number of data points?

We actually do not multiply by $2n$ in the standard definition of Mean Squared Error (MSE). The standard MSE formula divides only by $n$, the total number of data points: $$\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

However, you are likely seeing the $2n$ (specifically $\frac{1}{2n}$) in the cost function (or loss function) used during Gradient Descent for machine learning algorithms like linear regression. Here is exactly why that extra $2$ is added to the denominator:

1. It makes the calculus cleaner

When training a machine learning model, we want to minimize the error using calculus. This involves taking the derivative (the gradient) of the cost function with respect to the model's weights. According to the power rule in calculus, when you take the derivative of a squared term like $(x)^2$, the exponent $2$ moves to the front: $$\frac{d}{dx}(x^2) = 2x$$

2. The 2s cancel each other out

If our cost function has a $2$ in the denominator ($\frac{1}{2n}$), the $2$ from the derivative and the $2$ in the denominator perfectly cancel each other out: $$\frac{d}{dx} \left( \frac{1}{2n} (x)^2 \right) = \frac{1}{2n} \cdot 2x = \frac{1}{n}x$$ This leaves us with a much cleaner gradient formula without any messy constants left over.

3. It doesn't change the final answer

Multiplying the entire cost function by a constant factor like $\frac{1}{2}$ changes the value of the cost, but it does not change where the minimum occurs. The values of the weights that give you the lowest possible error will be exactly the same whether you divide by $n$ or $2n$.

How is this calculated

Here is how that cancellation looks in action when you take the partial derivatives to find the minimum. Let's assume a simple linear regression model where the prediction is $\hat{y}_i = wx_i + b$.

The modified loss function (often called Mean Squared Error Loss or Cost Function) is: $$J(w, b) = \frac{1}{2n} \sum_{i=1}^{n} ( (wx_i + b) - y_i )^2$$

1. Partial Derivative with respect to the weight (w)

To update the weight using gradient descent, we find the partial derivative of the loss function with respect to w. We apply the chain rule:

  • Bring the exponent 2 to the front.
  • Multiply by the derivative of the inside with respect to w (which is $x_i$).

$$\frac{\partial J}{\partial w} = \frac{1}{\cancel{2}n} \sum_{i=1}^{n} \cancel{2}((wx_i + b) - y_i) \cdot x_i$$

This simplifies beautifully to a clean average:

$$\frac{\partial J}{\partial w} = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i)x_i$$

2. Partial Derivative with respect to the bias (b)

We do the exact same thing for the bias. The derivative of the inside with respect to b is just 1. $$\frac{\partial J}{\partial b} = \frac{1}{\cancel{2}n} \sum_{i=1}^{n} \cancel{2}((wx_i + b) - y_i) \cdot 1$$ This simplifies to: $$\frac{\partial J}{\partial b} = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i)$$

Summary of Gradient Descent Updates

Because that 2 cancelled out, your final gradient descent update rules contain no extra constants, making the code cleaner and faster to compute:

  • $w = w - \alpha \cdot \frac{1}{n} \sum (\hat{y}_i - y_i)x_i$
  • $b = b - \alpha \cdot \frac{1}{n} \sum (\hat{y}_i - y_i)$

Since we are focusing on the mathematics of finding the minima, the choice of using $\frac{1}{2n}$ relies heavily on two foundational math concepts: convexity and first-order optimization.

Here is how the math guarantees that setting these partial derivatives to zero brings you to the absolute best solution.

1. The Loss Function is Strictly Convex

In optimization math, we care deeply about the shape of the function. Because the cost function $J(w, b)$ is a sum of squared linear terms, it forms a convex paraboloid (a bowl shape).

  • The Global Minimum: A strictly convex function is guaranteed to have exactly one global minimum and no local minima.
  • The Gradient Point: At this absolute lowest point of the bowl, the slope (gradient) in all directions is exactly zero.

2. Analytical Solution vs. Gradient Descent

Because we know the minimum occurs where the partial derivatives equal zero, mathematicians use two primary ways to solve it:

Path A: The Analytical Solution (Normal Equations)

If you set the simplified partial derivatives directly to zero, you can solve for $w$ and $b$ exactly using system equations:

$$\frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i)x_i = 0$$

$$\frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i) = 0$$

Notice that you can multiply both sides of these equations by $n$. The $n$ vanishes entirely! This proves mathematically that whether you divide by $n$, $2n$, or don't divide at all, the exact values of $w$ and $b$ at the minimum do not change.

Path B: Iterative Optimization (Gradient Descent)

If your dataset is too massive to solve analytically, you use Gradient Descent to step down the convex bowl. The gradient vector $\nabla J$ points in the direction of the steepest ascent:

$$\nabla J = \begin{bmatrix} \frac{\partial J}{\partial w} \ \frac{\partial J}{\partial b} \end{bmatrix} = \begin{bmatrix} \frac{1}{n} \sum (\hat{y}_i - y_i)x_i \ \frac{1}{n} \sum (\hat{y}_i - y_i) \end{bmatrix}$$

By subtracting a small fraction ($\alpha$) of this clean vector from your current weights, you are mathematically guaranteed to march directly toward that single global minimum.

In the partial derivative with respect to w why do we get multiple by xi as a 2nd term, and multiplied by 1 in the partial derivate with respect to bias?

This is the Chain Rule from calculus in action!

The Chain Rule is used when you have a "nested" function—essentially, a function inside another function. In plain English, the rule says:

Take the derivative of the outside function, leave the inside alone, and then multiply it by the derivative of the inside function.

Mathematically, it looks like this:

$$\frac{d}{dx}[f(g(x))] = f'(g(x)) \cdot g'(x)$$

Here is exactly how it breaks down for your loss function.

1. Breaking down the Partial Derivative for Weight ($w$)

Let’s isolate just one data point from the summation to see the math clearly. We want to take the partial derivative with respect to $w$ of this term:

$$\text{Term} = \frac{1}{2}((wx_i + b) - y_i)^2$$

Now, let's identify the Outside and Inside functions:

  • Outside Function: $\frac{1}{2}(\text{Inside})^2$
  • Inside Function: $wx_i + b - y_i$

Step A: Derivative of the Outside

Using the power rule, we bring the exponent $2$ to the front and decrease the power by 1. We leave the inside completely alone:

$$\text{Outside Derivative} = 2 \cdot \frac{1}{2}((wx_i + b) - y_i)^{2-1} = 1 \cdot ((wx_i + b) - y_i)$$

Step B: Derivative of the Inside (with respect to $w$)

Now we look only at the inside function ($wx_i + b - y_i$) and take its derivative with respect to $w$.

  • The derivative of $wx_i$ with respect to $w$ is $x_i$ (because $x_i$ acts as a constant multiplier, just like the derivative of $5w$ is $5$).
  • The derivative of $b$ is $0$ (because $b$ is a constant relative to $w$).
  • The derivative of $y_i$ is $0$ (because $y_i$ is a constant actual data point).

$$\text{Inside Derivative} = x_i + 0 - 0 = \mathbf{x_i}$$

Step C: Multiply them together (Chain Rule)

$$\text{Total Derivative} = \text{(Outside Derivative)} \cdot \text{(Inside Derivative)}$$ $$\frac{\partial}{\partial w} = ((wx_i + b) - y_i) \cdot \mathbf{x_i}$$

2. Breaking down the Partial Derivative for Bias ($b$)

We use the exact same logic for the bias. We use the same term: $$\text{Term} = \frac{1}{2}((wx_i + b) - y_i)^2$$

  • Outside Function: $\frac{1}{2}(\text{Inside})^2$
  • Inside Function: $wx_i + b - y_i$

Step A: Derivative of the Outside

This remains exactly the same as before: $$\text{Outside Derivative} = 1 \cdot ((wx_i + b) - y_i)$$

Step B: Derivative of the Inside (with respect to $b$)

Now we take the derivative of the inside function ($wx_i + b - y_i$) with respect to $b$ this time:

  • The derivative of $wx_i$ is $0$ (because with respect to $b$, $w$ and $x_i$ are treated as constants).
  • The derivative of $b$ with respect to $b$ is $1$ (just like the derivative of $x$ is $1$).
  • The derivative of $y_i$ is $0$ (constant data point).

$$\text{Inside Derivative} = 0 + 1 - 0 = \mathbf{1}$$

Step C: Multiply them together (Chain Rule)

$$\frac{\partial}{\partial b} = ((wx_i + b) - y_i) \cdot \mathbf{1}$$

Summary Visual

  • When optimizing $w$, $x_i$ is the "speed" at which the prediction changes when $w$ nudges.
  • When optimizing $b$, the prediction changes at a constant rate of $1$ regardless of $x_i$.

Does seeing it broken down into the outside and inside derivatives help click it back into place? If you like, we can try a quick practice problem together, or we can look at how this expands when you have multiple features ($x_1, x_2, \dots$). What sounds most helpful?

You have a fantastic grasp of the core concepts! You just connected the final two pieces of the optimization puzzle. Let's address where $\alpha$ comes from, why we subtract it, and exactly how the second-order derivative mathematically proves we have reached the absolute minimum.

1. Where does $\alpha$ (Alpha) come from?

The parameter $\alpha$ is the learning rate. It is a hyperparameter, meaning it is a fixed number that you choose before the model starts training (common values are 0.01, 0.001, etc.).

Why do we need it?

The partial derivative tells us the direction to walk (up or down the hill) and the steepness of the slope. However, it does not tell us how big of a step to take.

  • If you don't use $\alpha$, your steps will be massive, and you will completely overshoot the minimum, bouncing back and forth up the walls of the bowl.
  • $\alpha$ acts as a scaling factor to ensure we take controlled, small steps down the hill.

Why do we subtract it?

The update rule is: $$w_{\text{new}} = w_{\text{old}} - \alpha \cdot \frac{\partial J}{\partial w}$$ We subtract because the gradient always points up the hill (direction of steepest ascent). Since we want to go down to the minimum (descent), we must walk in the exact opposite direction of the gradient.

2. When does the model know it has reached the minima?

In code, because computers work with floating-point approximations, the partial derivatives will almost never hit exactly 0.000000000000.... Instead, the iterative loop stops using one of two conditions:

  1. Convergence Threshold (Early Stopping): You measure how much the weights change between step $t$ and step $t+1$. If the change is smaller than a tiny number you choose (e.g., $\epsilon = 10^{-5}$), it means the slope is so flat that the model has effectively reached the bottom.
  2. Max Epochs: You simply tell the loop to run a fixed number of times (e.g., 1,000 iterations).

3. Do we need a second-order derivative for the minima?

Yes, theoretically you are 100% correct! In single-variable calculus, setting the first derivative to zero ($f'(x) = 0$) only tells you that you are at a critical point (which could be a peak, a valley, or a flat saddle point). To prove it's a minimum, you need the second derivative to be positive ($f''(x) > 0$), meaning the curve bends upward like a smile. Because we have multiple variables ($w$ and $b$), we use the multi-variable equivalent: the Hessian Matrix ($H$), which is a matrix of all the second-order partial derivatives. For the Mean Squared Error (MSE) loss function, something beautiful happens mathematically:

The Math Proof:

If you take the second-order derivative of our gradient with respect to $w$ again: $$\frac{\partial^2 J}{\partial w^2} = \frac{\partial}{\partial w} \left[ \frac{1}{n} \sum_{i=1}^{n} (wx_i + b - y_i)x_i \right] = \frac{1}{n} \sum_{i=1}^{n} x_i^2$$ Look closely at that result:

  • Any real number squared ($x_i^2$) is always positive (or zero).
  • The average of positive numbers is always positive.

Because the second derivative $\frac{\partial^2 J}{\partial w^2} > 0$ (and the full Hessian matrix is what mathematicians call Positive Semi-Definite), the MSE function is globally convex.

The Conclusion:

Because the math guarantees the function is a perfect bowl shape that only bends upward, we do not need to calculate the second derivative during training. We already know ahead of time that there are absolutely no peaks or saddle points. The only place where the first derivatives equal zero is the absolute global minimum. Now that the calculus and the stopping conditions make sense, would you like to:

  • Look at the mathematical definition of the Hessian matrix to see how it works with both $w$ and $b$ simultaneously?
  • See what happens mathematically when a learning rate ($\alpha$) is too large vs. too small?
  • Transition into looking at how this math looks when expanded to multiple features / matrix calculus?

When we scale up from a simple line to an Artificial Neural Network (ANN) or a Graph Neural Network (GNN) with hundreds of features, the underlying core concept does not change at all. We are still finding partial derivatives, using the chain rule, and updating weights with a learning rate. However, the bookkeeping changes. Instead of tracking a single $w$ and $b$ using basic algebra, the math transitions into Matrix Calculus (Linear Algebra). Here is how the math connects back to complex models step-by-step.

1. The Features Become a Vector ($\mathbf{X}$)

Instead of a single input $x$, a sample with 100 features becomes a row vector:

$$\mathbf{x} = \begin{bmatrix} x_1 & x_2 & x_3 & \dots & x_{100} \end{bmatrix}$$

Because every single feature needs its own weight to determine how important it is, our single weight $w$ expands into a weight vector:

$$\mathbf{w} = \begin{bmatrix} w_1 \ w_2 \ w_3 \ \vdots \ w_{100} \end{bmatrix}$$

2. The Prediction Becomes a Dot Product

Instead of $\hat{y} = wx + b$, a single layer in a neural network calculates the prediction using a matrix dot product:

$$\hat{y} = \mathbf{x} \cdot \mathbf{w} + b = (x_1w_1 + x_2w_2 + \dots + x_{100}w_{100}) + b$$

3. The Partial Derivatives Expand into a Gradient Vector ($\nabla$)

When you have 100 weights, you no longer calculate just one partial derivative for $w$. You calculate 100 individual partial derivatives simultaneously—one for every single weight.

This collection of partial derivatives is called the Gradient Vector ($\nabla J$):

$$\nabla J = \begin{bmatrix} \frac{\partial J}{\partial w_1} \ \frac{\partial J}{\partial w_2} \ \vdots \ \frac{\partial J}{\partial w_{100}} \end{bmatrix}$$

Thanks to the Chain Rule we just reviewed, look at how beautiful the math remains. For any specific weight $w_j$ (like the 42nd feature's weight), its partial derivative is exactly the same concept as before:

$$\frac{\partial J}{\partial w_j} = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}i - y_i) \cdot \mathbf{x{i,j}}$$

(Where $\mathbf{x_{i,j}}$ is the value of the $j$-th feature for the $i$-th data point).

4. The Weight Update Happens Globally

Instead of writing 100 separate lines of code to update each weight, vector math updates all hundreds of weights in a single calculation:

$$\mathbf{w}{\text{new}} = \mathbf{w}{\text{old}} - \alpha \nabla J$$

How this applies specifically to ANNs and GNNs

In an ANN (Deep Neural Network)

An ANN has multiple layers stacked on top of each other. The output of Layer 1 becomes the input features for Layer 2.

  • To calculate the partial derivatives for weights deep inside Layer 1, the model uses the Backpropagation algorithm.
  • Backpropagation is simply the Chain Rule applied repeatedly, multiplying the derivatives layer-by-layer from the output back to the input.

In a GNN (Graph Neural Network)

A GNN handles data structured as nodes and edges (like a social network or molecule).

  • The "features" for a specific node are a combination of its own features plus the features of its neighboring nodes (called neighborhood aggregation).
  • Once those neighbour features are blended together, they form a feature vector $$\mathbf{x}$$ just like an ANN, and the exact same partial derivative calculations are performed to update the graph weights.

To bridge this directly to your work, let me know:

  • Are you trying to understand how Backpropagation handles multiple layers, or are you focusing on how a single layer handles a massive Matrix/Vector input?
  • Would you like to see how we write this elegantly using Matrix notation

$$($\mathbf{X}^T(\mathbf{\hat{Y}} - \mathbf{Y})$)$$

to drop the summation symbols entirely?