Homework 08

Read Sections 3.4 through Section 3.5 of the Raschka LLM book (the middle part of Chapter 3).

Create a directory called week8 in your AI assignments directory and commit your work for today there.

<repo_dir>/ai-24sp/assignments/<your_username>/week8

Create empty files in this directory. Read through and type the code in each section and add your work in the appropriate filename. Run your code with a different random seed than the one shown in the book

torch.manual_seed(456)

• 3_4_1_manual_attention.py
• 3_4_2_compact_class.py
• 3_5_1_causal_mask.py
• 3_5_2_dropout.py
• 3_5_3_causal_class.py

Commit and push using our GitWorkflow.

Add an entry to your dev diary containing your answers to the following questions and outputs of running your code above.

In Section 3.5.3, there is a code listing for a compact self-attention class.

• On line 2 are the parameters to the constructor.
  • When we call this class later in the section, what exact numbers do we give for each parameter value?
    • d_in
    • d_out
    • context_length
    • dropout

Give the values of the variables on line 16:

• b
• num_tokens
• d_in

Give the shape of the variables on lines 18-20:

• keys
• values
• queries

Give the shape of the variable on line 22

• attn_scores

and line 25

• attn_weights

Use Python3's print function in your 3_5_3_causal_class.py to verify your answers, and copy and paste the output into your dev diary.

In Section 3.5.4, when you are working on multi-headed attention:

Change the input arguments for the MultiHeadAttentionWrapper(..., num_heads=2) call such that the output context vectors are 2-dimensional instead of 4-dimensional while keeping the setting num_heads=2. Hint: You don't have to modify the class implementation; you just have to change one of the other input arguments.

Run and test your work in 3_4_3_multi_head.py.

Read pages 1-7 and the last paragraph on page 8, of Ethics and Second-Order Cybernetics.

In your dev diary, write an essay responding to one or more of the following prompts:

• What is cybernetics?
• What is the relationship of cybernetics to artificial intelligence?
• What is ethics...
  • ...in your personal understanding before reading this essay?
  • ...as described in this essay?
  • As far as your personal view is similar to or different from the meaning of ethics implied in the essay, say more about how your view developed
• What is the cybernetics of cybernetics?
• What kinds of non-verbal experiences is it possible to have with a large language model that is focused mostly on verbal interactions?
  • For non-verbal experiences which are currently impossible to have with an LLM, do you think it will ever be possible, and how will you know?
• How can cybernetics affect artificial intelligence, in particular language interactions and recent LLM progress?
• How can artificial intelligence, in particular LLM progress, affect cybernetics?