wshine ai hw8 - TheEvergreenStateCollege/upper-division-cs-23-24 GitHub Wiki

AI Homework 8

Questions

  1. d_in : 3

    d_out: 2

    context_length: 6

    dropout: also not used as a parameter to the constructor. but the book had us using a dropout value of 0.

  2. b : 2

    num_tokens: 6

    d_in : 3

  3. each of keys, queries, and values are 2x6x2

  4. 2x6x6

  5. 2x6x6

2 6 3
keys:  torch.Size([2, 6, 2])
queries:  torch.Size([2, 6, 2])
values:  torch.Size([2, 6, 2])
attn:  torch.Size([2, 6, 6])
attn:  torch.Size([2, 6, 6])
context_vecs.shape: torch.Size([2, 6, 2])

unsure what to do for this exercise. we are asking to modify the parameters so the shape of the context vector reurned is two-dimensional instead of 4, but the it currently has a shape of 2x6x2.

current ouput with no modifications to parameters:

tensor([[[0.3190, 0.4858],
         [0.2943, 0.3897],
         [0.2856, 0.3593],
         [0.2693, 0.3873],
         [0.2639, 0.3928],
         [0.2575, 0.4028]],

        [[0.3190, 0.4858],
         [0.2943, 0.3897],
         [0.2856, 0.3593],
         [0.2693, 0.3873],
         [0.2639, 0.3928],
         [0.2575, 0.4028]]], grad_fn=<ViewBackward0>)
context_vecs.shape: torch.Size([2, 6, 2])
⚠️ **GitHub.com Fallback** ⚠️