Challenge #6 - ajainPSU/ECE410-510-Codefests GitHub Wiki

Overview

The objectives of the challenge was to implement a simple perceptron neuron with two inputs and a sigmoid activation function. From which we were to train it to recognize via the perceptron learning rule to realize the NAND and XOR binary functions.

The sigmoid function as Saeed states is that it is a special form of the logistic function, with an S-shaped curve. The function itself is continuous everywhere and monotonically increasing, and is also referred to as a squashing function since it's range is either zero or one, making it ideal for binary functions. In the realm of neural networks, as this challenge/exercise was posed, it's used as a activation function, where weighted sums of inputs are passed through it and the output is presented, which is used as an input to the next layer. The sigmoid function as an activation function for a neuron (perceptron in this case) will assure that the output will always be between 0 or 1.

Now in the realm of the Perceptron Algorithm it lists as according to Dukor:

Initialize weight values and bias
Forward propagate
Check the error
Back propagate and adjust weights and bias
Repeat for all examples.

In which it's an algorithm under the supervised learning category for binary related classifications, a linear classifier. Now for the sigmoid it's also non-linear, for if it was linear then it could only learn anything that was linearly separable. With it being non-linear, then the sigmoid function can be put in a hidden layer to solve non-linear problems.

Lastly I acknowledge the use of LLM's for this, specifically Co-Pilot as it assisted with developing and explaining the neuron.

Initial Perceptron

The initial task is to develop a perceptron that has two inputs with the sigmoid activation function. Since I was lost on where to start or how lengthy such a script would be in python, I asked my LLM (Github Copilot) to show me an example in which I used as a start point. Figure 1 below shows my inquiry and result. Note that I didn't test the perceptron itself, but rather used it to see how it was built into when training for NAND and XOR.

Fig111 Figure 1: Inquiry & Result from LLM on a Simple Perceptron.

Perceptron - NAND

Moving from the initial Perceptron, which is recorded as the file Perceptron.py in the Challenge #6 folder, the next step was to train this simple neuron to recognize the NAND function. Figure 2 below shows a simple two-input NAND function's truth table.

Figure 2: NAND Truth Table

In order to expedite the challenge's progress, additional "vibe coding" was used where the LLM was inquired to adjust the initial perceptron code's to be able to train the NAND gate. Figure 3 below shows the inquiry and the resulting software it responded with.

Fig222a Fig222b Figure 3: LLM Inquiry and resulting software to train perceptron for NAND function.

The file that has the python script for the NAND training is listed as Perceptron-NAND.py in the Challenge 6 folder. After incorporating the LLM's insights into the base perceptron, the script was run and trained, in which Figure 4 and 5 show the results in the command terminal of the end results -- a successful training, and the training error over the epochs, in which over time as the epochs increased, the error decreased.

nandresult1

Figure 4: Console output of the Perceptron-NAND training, confirming it realized the NAND function.

nandresult2

Figure 5: Graphical representation of the neuron conducting less error as it was trained to realize the NAND function.

Perceptron - XOR

The other function that the Perceptron was to be trained and realized is XOR, however before I actually did XOR separately, I tried and failed to incorporate the perceptron to train and learn both NAND and XOR at the same time, which the file for that is labeled Perceptron3.py in the folder -- see Learning Takeaways for more. Below is Figure 6, the XOR truth table.

xor-gate-truth-table (1)

Figure 6: XOR truth table.

Figure 7 shows the inquiry made to the LLM for just the XOR training, which resulted in the file Perceptron-XOR.py. Figure 8 shows the final result in the console with the perceptron realizing the XOR function, and Figure 9 shows the training error decreasing over time via the amount of epochs, which was increased to 50k due to it requiring more to properly realize. Additionally, XOR cannot be realized without the perceptron being a multi-layered perceptron, since XOR itself is a non-linear function.

7aa

7ab

7ac

Figure 7: LLM Inquiry & Software result for the Perceptron to be trained and realize XOR.

xor-1

Figure 8: Console results showing the perceptron correctly trained to realize the XOR function.

xor-2

Figure 9: Graph of errors decreasing over epochs as the perceptron learns XOR.

Learning Takeaways

The main takeaway I learned from this, was that the perceptron was only able to really to learn one function at a time. Before I got the XOR function to work, I tried having it learn and realize both NAND and XOR at the same time. This didn't work since by doing that, when the perceptron was trying to learn, it couldn't properly shift the weights, thus overriding them when switching functions to be trained. In addition, the weights would be optimized for one function, but not the other. This combined neuron training script is in the folder as Perceptron3.py. Figure 10 shows the LLM Inquiry made to attempt to combine the learning, Figure 11 shows the console output, Figure 12 shows the error over 10k epochs, and Figure 13 shows the error over 20k epochs.

10a

10b

`0x

10d

Figure 10: LLM Inquiry for a Perceptron to train both NAND and XOR at the same time.

combined

Figure 11: Console terminal output of the Perceptron attempting to learn both NAND & XOR at the same time.

10k

Figure 12: Training error over 10k epochs as the Perceptron attempts to learn both NAND & XOR at the same time.

20k

Figure 13: Training error over 20k epochs as the Perceptron attempts to learn both NAND & XOR at the same time.