Summary

The forward pass of the perceptron with two inputs (x1, x2) is simply prediction = sigmoid((x1 * w1) + (x2 * w2) + b). The goal is to learn the weights (w1, w2) and bias (b) from the data for a NAND and XOR truth table. This is achieved through perceptron learning, a form of gradient descent that updates the weights by propagating the error = true_value - prediction through to the weights using the following equation:

        delta = error * sigmoid_prime(output)
        w1 += learning_rate * error * x1
        w2 += learning_rate * error * x2
        b += learning_rate * error

Sigmoid prime is the derivative of the sigmoid function, which I derived below for fun.

I trained for 10,000 epochs, and you can see for the NAND gate that the predictions are correct with the inference values being very close to either 1 or 0:

Epoch: 0, Total Error: 1.291919063305945
Epoch: 1000, Total Error: 0.009485340294862072
Epoch: 2000, Total Error: 0.002505661446126139
Epoch: 3000, Total Error: 0.0011243887732264573
Epoch: 4000, Total Error: 0.0006336612524775838
Epoch: 5000, Total Error: 0.00040546975851882673
Epoch: 6000, Total Error: 0.00028134603261309595
Epoch: 7000, Total Error: 0.00020649703010294527
Epoch: 8000, Total Error: 0.00015793907957368136
Epoch: 9000, Total Error: 0.0001246705296331466
0 NAND 0 = 1 (raw: 1.0000)
0 NAND 1 = 1 (raw: 0.9950)
1 NAND 0 = 1 (raw: 0.9949)
1 NAND 1 = 0 (raw: 0.0071)

For the XOR data, which cannot be learned with a perceptron, the data was not correctly predicted for (0,1) and (1,1). It's clear from the decrease in error in both cases that there was no need to train for this long:

Epoch: 0, Total Error: 1.1433136746601436
Epoch: 1000, Total Error: 1.0519276942900175
Epoch: 2000, Total Error: 1.0519276944175484
Epoch: 3000, Total Error: 1.0519276944175486
Epoch: 4000, Total Error: 1.0519276944175486
Epoch: 5000, Total Error: 1.0519276944175486
Epoch: 6000, Total Error: 1.0519276944175486
Epoch: 7000, Total Error: 1.0519276944175486
Epoch: 8000, Total Error: 1.0519276944175486
Epoch: 9000, Total Error: 1.0519276944175486
0 NAND 0 = 0 (raw: 0.4872)
0 NAND 1 = 0 (raw: 0.5000)
1 NAND 0 = 1 (raw: 0.5128)
1 NAND 1 = 1 (raw: 0.5256)

The Colab notebook with full code is available in my 510 folder.