Starting this wiki page to add entries. Stay tuned for more edits.

11-APR-2024

Reading Response:

This article was centered around the idea of whether or not AI image generation is ethical/permissible to use for someone posthumously. Arguments can, and have been made for either side. With the idea that one is preservation, and the other is falsifying things that that person did not do in actuality.

While I find the technology to do these things fascinating, I do think it raises an ethical dilemma; if we can digitally reconstruct a person, and program their moves and words as a digital animation, where it looks virtually indistinguishable from the person in real life, this can be used as a manipulation tactic.

Of course one can say that this is actually preservation of a person, and having them digitally reconstructed is almost a way of keeping them alive forever. Immortalised in a digital monument of sorts.

There really seems to be a fine line, between preservation, and exploitation in this case.

25-APR-2024

AI Ethics Questions:

Ethics around digital voice reconstruction are always a tricky topic. Obviously, if someone give explicit consent and/or is compensated, it should be 100% allowed to clone someones voice. It's when they don't give explicit consent that it could become an issue. I don't think it is ethical to synthesise someone else's voice without direct consent, but I also don't think that it is ethical to legally regulate this. When it comes to these digital tools, I honestly do not believe that it is ever ethical to over regulate what someone can do with technology, so obviously I believe that everyone should have the right to clone their own voice, and within a grey area, the right to clone any voice.

This idea is in a paradox for me, because I think there are controversial and unethical ideas on both sides.

AI Voice Training Notes:

We (@faulkdf and @dawsonwhi) used a ~5 minute audio clip of me talking. I read the first few paragraphs of Moby Dick. The output audio was clear and understandable, but I was maybe somewhat relived to hear that it didn't quite sound like me. It certainly sounded like an adult male, about 20-30 years old with a Pacific Northwestern American accent, but it wasn't an exact copy of me. I was impressed that our clip sounded as good as it did, seeing that I only gave it about 5 minutes of data to work with. This Lab didn't change my ethical ideas about this, I still have mixed feelings about synthesising voices, but I enjoyed trying to clone mine. One thing I would like to better understand, is are there diminishing returns on how much data you fine-tune/train the model with? Does it get to a point where more data isn't helpful? Or is more data always better?

4-MAY-2024

This is AI HW 3 that I actually started a while ago, just now got around to adding it here:

AI HW 3:

AI Tech Reading:

1: Connectionism is a theory that says complex ideas emerge from interconnected networks of simple parts.In MNIST, this means recognising patterns like circles in different digits, like the top of 9 or the top/bottom of 8, by learning to identify features like curves and lines spread across the network. This typically helps a neural network generalise its understanding of what it knows.

2: One of the biggest things that has been a big influence on deep learning recently is computer infrastructure, (both hardware and software) has improved immensely in the last few years. The increased dataset sizes has also had a profound effect.

3: According to Fig. 1.11, humans have on average 10^11 neurones, whereas, according to the same figure GoogLeNet had between 10^6 and 10^7 neurones simulated. 
Today, I asked OpenAI’s chatGPT, and I conducted a regular search query. chatGPT wouldn’t give me a straight answer, it would just resort back to Petaflops achieved by various super computers. My google query found a result saying that sometime this year, a computer called “DeepSouth” will boot up, and be capable of “228 trillion synaptic operations per second” (Source: https://www.newscientist.com/article/2408015-supercomputer-that-simulates-entire-human-brain-will-switch-on-in-2024/) I was having a lot of trouble finding a number for currently used neural networks, as all of the sources claimed different numbers were the biggest.

4: The reason for this behaviour lies in the fact that the neural network has not yet been trained to recognise patterns in the input data and make accurate predictions. Before training, the network's parameters are essentially random, so its output is unrelated to the input data. As training progresses and the network learns from the data, its predictions gradually become more accurate and start to resemble the true labels.

5: The shape tuple for the neural network's input is (7, 13, 2).

6: I think this is right: self.weights: Shape between the first and second layers: (13, 7) Shape between the second and third layers: (2, 13) self.biases: Shape for the biases in the first layer: (13, 1) Shape for the biases in the second layer: (2, 1)

7: I could not get the image to load in any browser I tried, therefore I could not complete this question.

8: Changing the learning rate can have a whole array of changes to a neural network, such as the convergence speed, which can lead to faster convergence, but if the learning rate is too high it can overshoot the target.

9: The term "stochastic" in "stochastic gradient descent" refers to the fact that the gradient used to update the model parameters is estimated using a single randomly chosen data point from the training set, rather than using the entire dataset as in traditional gradient descent.

Human Writing:

1: Slide Prompt:
ChatGPT can be a very useful tool, but as it currently stands, it is a very advanced autocorrect. It is not inherently intelligent, and cannot generate writing better than a well researched and informed human. 
It can, however, generate something somewhat decent, within seconds. When ChatGPT was in the news and people were really impressed/scared of its capabilities, I reused some old essay prompts from when I was completing my associates degree, and compared its result to mine (that I got a good grade on). Predictably the result was not better. But it was still impressive that it generated something legible in seconds. 
ChatGPT has been very useful to me for some automation and styling for both code and documents. I have used it for style guides when writing in markdown, and I have asked it to attempt to make my code more streamlined with better formatting. ChatGPT is, a useful tool, that can be adapted to many uses. But in the end it is that; a tool. It will not replace humans in its current state.

2: Framing in journalism refers to the way in which news stories are presented to the audience, emphasising certain aspects while downplaying or omitting others. It involves selecting certain facts, themes, or interpretations to shape the audience's perception of an issue or event.

3: System Prompts are essentially journalistic framing for an LLM. They are designed to steer the AI in a specific direction to accomplish a goal under specific pretences.

4: If I understand the question correctly, the “standard story” is so prevalent in our media, that this is going to be the primary thing an AI is trained on. LLMs can only be a reflection of the information we give them. So naturally when you might ask one to give you a story, it will print out something with the “standard story” sort of arc.

5: Research assistance powered by an AI assistant can be useful in interpreting a human command, and finding sources and explaining concepts. I feel like it can be used as a tool to boost productivity, but it can also be relied on too heavily and cause the writing quality to suffer. If used correctly it can be a very useful tool.

10-MAY-2024

AI HW 5

Chapter 1:

1: GPT and LLM are terms that are often used interchangeably. GPT is Generative Pre-trained Transformer and LLM is Large Language Model. The terms are synonymous.

2: A, C, D for sure, because the nature of forums is that they are usually question and answer based. I could see this working for B (Twitter) but typically that is not how the platform is used.

3: Attention is All You Need was originally designed to translate human languages.

4: 3 or more layers is considered deep learning.

5: yes

6: True and False Questions:

A. True: Pre-training GPTs on a large corpus of text data can take weeks or even months, whereas fine-tuning for a specific task might take much less time.

B. False: Pre-training for GPT models is typically done on large amounts of unlabelled text data, while fine-tuning often requires labeled data for specific tasks such as sentiment analysis or text generation.

C. True: A research team might pre-train a GPT model, and then another team in a different organisation can fine-tune the model for a particular application or domain.

D. True: Pre-training aims to impart broad knowledge to the model, making it suitable for various tasks, while fine-tuning adapts the model's parameters to a specific task, making it more specialised.

E. True: The fine tuning dataset, while large, is often smaller than the enormous amount of data used for training.

F. True. Pre-training involves training the model from scratch on a large corpus of data, while fine-tuning involves adjusting the parameters of a pre-trained model to fit a specific task or domain.

7: All of these are true except for E. A-D can all frame how a GPT predicts the next word in the sentence.

8: Here’s my chat with OpenAIs chatGPT: https://chat.openai.com/share/7a43c6fa-71c2-40cb-9ab4-2f8db71b819d

9: The Encoder and Decoder are both present in neural networks. They are essential for computing data in a higher level, and bringing it back down to be human readable. Tokenising is not present in neural networks, however it is essential in the training process.

10: chatGPT and our TTS model are examples that have zero-shot. MNIST and Outlook filters both have very specific training data that it is working off of. Neither anticipates seeing something its never seen, whereas GPT and our TTS both get fed new information all the time.

11: In zero-shot learning, the model is trained to recognise classes or perform tasks for which it has not seen any labeled examples during training. It pulls from its vast “knowledge” and generalises a response. 
In few-shot learning, the model is trained on a very small amount of labeled data for each class or task. The model learns to generalise from this limited labeled data to perform tasks on new, unseen examples. Many-shot learning is the traditional learning paradigm where the model is trained on a large amount of labeled data for each class or task. Many-shot learning is commonly used in scenarios where abundant labeled data is available for training, like MNIST.

12: GPT-3 and GPT-3.5 have been trained on 175 Billion Parameters. According to this article, OpenAI hasn’t revealed how may parameters GPT-4 has, but Andrew Feldman, CEO of AI company Cerebras, says that GPT-4 has perhaps 100 trillion parameters.

Chapter 2:

1: Large Language Models (LLMs), such as GPT, can't operate on words directly because they require a numerical representation of the input text to perform computations. Just like in MNIST, where input data (images of handwritten digits) were converted into numerical representations before being processed by the neural network, text data must also be converted into numerical form for LLMs to work with. Im not sure if this is exactly right, but computers tend to prefer math, and words aren’t intrinsically numeric.

2: An embedding is a representation of an object or entity, such as a word, in a lower-dimensional vector space. The dimension of an embedding refers to the length of the vector representing the word in the vector space.

3: I believe the short story is within a 256 dimension of embedding. GPT-3 is 1,024.

4: The correct order for processing and preparing the dataset for LLM training is: C. Breaking up natural human text into tokens. B. Giving unique token IDs to each token. A. Adding position embeddings to the token word embeddings. D. Converting token IDs to their embeddings.

Human Writing:

The question about creativity I find very interesting. “What is creativity” is such a good question, because it is such a broad definition. I think creativity is something that is inherently human/intelligent, and until we have a true artificial intelligence that is capable, all of the “AI” that we have isn't properly sentient or intelligent.

Art that is produced through generative AI rides a very fine line on whether I consider it creative or not. (Remember this is purely my opinion.) I think there are use cases, where using generative AI is creative. I saw one piece that an artist made where they asked an LLM for a description of a cat, without using words that typically describe a cat (like feline, animal etc). They took the words the LLM generated, and used it as a prompt for a text-to-image generative AI. The images it produced, barely looked cat-like, yet there was something that did perfectly match the prompt. I thought this idea was creative, and it created these abstract pieces that tell a unique story, on how AI scripts interact with each other.

However There are ways that generative AI use is not creative, and it is purely lazy and profit driven. Such as creating massive amounts of designs for marketing, using LLMs to write articles for a publication, or flooding a social media algorithm with content faster than a human could produce it. Uses of generative AI in these spaces, take over areas that have always had a human on the other side. I am a musician, creativity is very important to me. I want my music to be made by a person. I want it to be an expression of their artistic talent, no matter the genre. If the music is AI generated, its just a predictive amalgam of everything that it was trained on. While technologically impressive, it lacks the very thing that makes music enjoyable: a feeling. A feeling that someone, with a life, and emotions and with thoughts and memories can only make.

I don’t think that generative AI should be banned in any way. I am very against regulation of things of this nature (plus what would the logistics of this even be). I still however am not a fan of art being made by software, and only by software. Once an AI is properly intelligent, and sentient, it might be a different story. As of now, however, every AI script is basically a glorified autocomplete: “completing” data from a prompt a user entered. While it may look like new content, be it written, sonic, or visual, it technically isn't. It is filling in the gaps from its vast array of data to generate something that seems new, but its really just everything it has been trained on regurgitated in a seemingly organised fashion.

Overall, I think creativity is a trait that is exclusively sentient. By the time AI is able to catch up to us, and be sentient. There won’t be a need for humans on this planet anymore.

21-MAY-2024

Sidenote: I may have to figure out a more organised way to do this dev-diary. It is starting to become a mess just adding entries sequentially

SC HW Quiz Screenshots:

Chapter 10:

Q1-1:

3 seemed like an obvious answer to me for this one

Q1-2:

this snippet of program didn't seem complete enough to compile correctly to me.

Q1-3:

This also didn't seem like it would compile correctly to me.

Q2-1:

The syntax to me looked correct off the bat, but it took some thinking as to what the output would be. Since Dog was the only one defined with a noise, it made sense that bark would come before (silence)

Q2-2:

I was unable to figure out the correct answer and to be honest the correct one doesn't even make sense to me.

Q3-1,2:

1: This didn't seem like it would compile 2: I actually cant remember my reasoning for this one.

Q4-1,2:

1: I also can't remember my reasoning for this answer as well 2: the program didn't look complete

Q5-1,2,3:

1,2,3: I had a lot of trouble with this one and on numerous attempts I didn't get it right. There is something I didn't understand.

Q6-1,2,3,4,5,6:

1: The first one seemed obvious to me, that you cannot elms as mutable 2: They all didn't seem complete 3: This chunk seemed to be the right one to me 4: This one also had to do with borrowing mutables. 5: None of these seemed correct either 6: This was one of the simplest answers and it looked like it would work to me.

Chapter 11:

Q1-1,2:

1: This just seemed obvious, the #[blank] format is how the annotations work in rust 2: The context explains it #[should_err] doesnt exist in rust.

Q2-1,2:

1: This one seemed like it could cause a race condition. 2: Upon seeing the answer I realised mine was stupid but I didn't think of it at the time and I didn't want to keep guessing.

Q3-1:

This one seemed redundant, as all tests have access to private functions.

Update on Game of Life

I had some good progress with the project this week, and I was able to get it to properly animate using canvas instead of unicode characters. The book is hard to follow but I managed to figure it out. Working on continuing to implement code from the book this week.

19-JUN-2024

The Last Two

AI HW 8:

Human Writing:

The relationship between cybernetics and artificial intelligence is deeply intertwined, with cybernetics serving as a foundational discipline that has significantly influenced the development and conceptualisation of AI. Both fields share a common goal of understanding and creating systems that can process information, learn, and adapt. However, they approach this goal from different perspectives.

Cybernetics provides the theoretical supports for understanding feedback mechanisms, control systems, and adaptive behaviour. These concepts are crucial for the development of AI, which seeks to create machines that can perform tasks requiring human-like intelligence. The feedback loops and adaptive processes studied in cybernetics are essential for developing AI systems capable of adjusting their behaviour based on new information and experiences.

Moreover, cybernetics has influenced the development of key AI technologies and methodologies. For example, the concept of feedback is central to the design of neural networks, which are computational models inspired by the structure and function of the human brain. Neural networks learn by adjusting the connections between nodes based on the feedback received during training. This process mirrors the self-regulating mechanisms studied in cybernetics.

Additionally, cybernetics' focus on the interaction between systems and their environments is reflected in AI's emphasis on creating intelligent agents that can perceive and act upon their surroundings. This is evident in fields such as robotics, where AI-driven machines must navigate and interact with complex environments, and in machine learning, where algorithms must adapt to changing data patterns.

Cybernetics provides a good baseline that we need to continue studying, in order to have a better concept of Artificial Intelligence as the field continues to evolve.

SC HW 8:

Rust Book:

I did fairly well on all of the quizzes in the rust book, I was just confused about this one, and could not figure out what the answer was supposed to be (even after consulting an AI)

GitHub isn't letting me upload images (I must have reached my limit) but I will copy out the question and my answer here:

Question 2: Determine whether the program will pass the compiler. If it passes, write the expected output of the program if it were executed.

fn main() {
    let mut s = String::from("Hello");
    let add_suffix = |s: &mut String| s.push_str(" world");
    println!("{s}");
    add_suffix(&mut s);  
}

I said that the program Does Compile (which the book says is true) and that the Output would be {s} (which the book says is incorrect)

I have no idea what else the output would be, unless somehow it is Hello World which doesn't make sense to me.

Game of Life:

I have a working Game of Life model that you can interact with. I should have pushed all of my code to the repo in the faulkdf folder somewhere within sc-24 Again, I cannot add a screenshot because I think I am maxed out, but if anyone wants one I can send one via email.

Thanks everyone for a great class!