Week 09 - Afternoon Lab

This is our last lab before our final project fair in Week 10 with CSF.

Our goals today:

• Moving ahead to Chapter 5 code, we will train our GPT on our chosen dataset
  • We will start with the seed phrase "Every effort moves you" at first as shown in class.
  • Adapt it to accept a custom chat prompt from you as a command-line argument
  • Loop indefinitely and
    • print the model-generated text to the user
    • accept new input from the user
    • keep submitting the entire current chat into your model with the new user input, and repeat
• Download and use the pre-trained OpenAI model, and compare with one that you train.

Don't worry if you haven't finished the previous lab. Start fresh with today's lab.

Change back to your assignments directory, one directory up from the data directory in the previous step.

cd <repo_dir>/ai-24sp/assignments/<your_username>

Download your dataset / book text using your data.sh script from last time, if you don't have your text in your current work environment.

(For example, a new GitPod workspace).

./data.sh

Create a new directory for week9 and do your work by adding and running the following files from the official Raschka github to this directory.

• previous_chapters.py
• gpt_train.py

mkdir -p week9
cd week9
touch dataloader.py

From yesterday's class, you'll need to change the appropriate line to load the source text file from a path that looks like

../data/mark-twain-autobio.txt

but for your chosen dataset.

You may also need to copy over your requirements.txt file from week7 or week8 which looks like

tiktoken
torch
numpy
matplotlib

Install these packages with this command

pip3 install -r requirements.txt

This may take a few minutes on GitPod, so start the process and read ahead to the next step.

Read Chapter 5 of the Raschka book on LLMs, starting from the beginning and up to and including Section 5.

Your goal is to adapt the files gpt_train.py and previous_chapters.py to measure

• Validation and training loss on your chosen dataset, reading Section 5.1
  • Compare your validation and training loss to that shown in class yesterday.
  • Does your validation loss remain higher than training loss? How can we close this gap?
• Read the section "5.4 Loading and saving model weights in PyTorch" to make sure you don't have to train each time
  • you may wish to have a command-line argument, or copy and paste a new python script that loads a model for inference instead of training it
• Adapt the Python code so you can pass it a command-line argument seed phrase to start a conversation
  • so you don't have to use the hard-coded phrase "Every effort moves you"
• Read Section "5.5 Loading pretrained weights from OpenAI" and try chatting with both the OpenAI model and your own model.
  • How are they different or the same?
  • Write your discussion into your dev diary.

In your assignments directory, run the following command to prevent yourself from accidentally committing any large files to the class monorepo.

Now add, commit, and push your changes on a branch and pull request using our Git Workflow.