Developing Locally Guide - brown-cs1690/handout GitHub Wiki

This year, we are introducing a new development environment for the class using a Podman/Docker container. This will allow you to develop fully locally, without having to connect to department machines. However, since this is the first time we are rolling out this setup, we are also including instructions to develop on department machines, in case anything goes wrong with the container. This guide will show you how to set up both of these environments.

Disclaimer: This guide looks daunting, but rest assured that you will only need to set this up once. If you encounter any issue, please post on Ed, talk to us in office hours, or reach out to your mentor! (more on this soon)

To run our environment, there are three main components you need to configure on your system:

• An X11 Server, which is a framework powering GUI applications in Linux. You will need this to view the Weenix terminal.
• Docker or Podman, which allows you to build and run the OS container, and run code in the course.
• Visual Studio Code, an IDE, which allows you to edit the code easily and incorporates lots of useful features such as IntelliSense.

Depending on your computer's OS, you will need to follow different instructions.

Shout out to the CSCI 1680 Staff who wrote this Container Setup guide, which helped a lot in writing this section of the guide.

1. Download and install XQuartz. Run the installer, and allow the app to make changes when prompted. You may need to log out and log back in to your system to apply changes.
2. Run XQuartz from the application menu. Once it opens, go to the menu bar at the top of your screen and select XQuartz > Preferences. Go to the Security tab and check the box labeled "Allow connections from network clients". This will allow the container to connect to your X server.
3. You can now quit XQuartz; it should re-open automatically when you run Weenix.

1. Download and install VcXsrv. Follow the instructions and use default settings.
2. There should be a new icon on your desktop "XLaunch". Click on it, and keep clicking Next in the bottom right of the window until the window disappears.
3. In the future, remember to run VcXsrv every time you need to run Weenix. You can do so by repeating step 2.

If you are using Linux, you most probably already have an X server installed. You should just ensure that xhost, a tool used to control access to the X server, is installed.

You can check that by running: which xhost

If the result is xhost not found, you will need to install this on your system. The command to do so varies across distributions; you can probably find the right one by Google-ing. If you need any help with this, feel free to post on Ed.

Whether you install Docker or Podman depends on your computer's architecture. On M1-M4 Macs, we've had better results at emulating x86_64 programs (such as Weenix) using Podman's virtual machines, as opposed to simply running a Docker container.

1. Head to the Podman website, click the Download button and click "Podman Desktop for macOS".
   
   
2. Open the .dmg file you downloaded and drag and drop the app in the Applications folder.
3. From the Applications folder, run Podman Desktop. If the app prompts you to install "Podman", accept to do so. You can skip any other extension that the app suggests you to get.
4. When the extension finishes downloading, click the bottom left "Settings" icon, and head to the "Preferences" section (last section in the settings menu). There, search for "Rosetta". Switch the toggle off.

1. Install Windows Subsystem for Linux (WSL) and the Ubuntu Linux distribution by opening PowerShell or Windows Command Prompt in administrator mode (right-clicking and selecting "Run as administrator") and running the wsl --install command. Then restart your computer.
2. Run wsl -l -v to see what distributions you have installed. If none go to the Windows store and install Ubuntu 24.04. If you encounter an error, make sure that the box Virtual Machine Platform is checked off within the Windows Features Control Panel
3. Install Docker Desktop for Windows.
4. Ensure the WSL 2 back-end is enabled: Right-click on the Docker taskbar item and select Settings. Check Use the WSL 2 based engine and verify your distribution is enabled under Resources > WSL Integration.

1. Install Docker Desktop for Mac

1. Follow the official install instructions for Docker CE/EE for your distribution. If you are using Docker Compose, follow the Docker Compose directions as well.
2. Add your user to the docker group by using a terminal to run: sudo usermod -aG docker $USER
3. Sign out and back in again so your changes take effect.

Install Visual Studio Code, then install the Dev Containers extension.

1. Find the directory where you want to store your coursework, and clone the os-devcontainer repository:
   git clone https://github.com/brown-cs1690/os-devcontainer.
2. Navigate to the directory you just cloned cd os-devcontainer. If on Windows, launch WSL with the wsl command.
3. Run cd container, then ./build-container. This command will build your Docker/Podman image, which may take a few minutes.
4. cd .., then ./run-container should run the container.

If you see something like cs1670-user@localhost:~$, you're good to go! You can now run any Linux commands from this command line. To exit the container, type exit or Ctrl-D.

You might have noticed a directory called home inside of os-devcontainer. This is where the container’s home directory is mounted to. That means any changes made in one will be reflected in the other.

Whenever you clone an assignment, either clone it into os-devcontainer/home, or your home directory inside of the container (/home/cs1670-user).

Tip: You can cache your GitHub credentials so you don’t have to enter them every time with the following command:
git config --global credential.helper store

If you haven’t already, start your container with ./run-container. This needs to be run from WSL if you are on Windows. Make sure the Docker Desktop app is running in the background.

A neat feature of the Dev Container extension in VS Code is that you can connect to the container's console, as if you were running VS Code on it directly. To do so:

1. Launch VS Code

2. If you are NOT using an M1-M4 Mac, ignore this step. Otherwise, you need to make sure the Dev Containers extension looks for Podman containers, instead of Docker containers (which is the default option):
   2.1. In the bottom left of the page, click the cogwheel, then head to "Settings".
   
   2.2. A window opens up. Look for "podman". In the "Dev › Containers: Docker Path" setting, change docker to podman. Note that you might need to revert this when developing for another class which uses regular Docker containers.
   2.3. Exit the settings

3. Click the blue arrows button at the bottom left, then select "Attach to Running Container…", and choose the os-devenv container.
   

4. Once it loads, hit File->Open Folder… and navigate to the assignment directory (e.g. /home/cs1670-user/uthreads-mgyee)
5. Start coding!

Tip: If you are trying to use VS Code’s C/C++ extensions (autocomplete, syntax highlighting, etc.) and are seeing the following error:
cpptools client: couldn't create connection to server

Run the following command from your home directory in the container:
sudo chmod -R 755 .vscode-server/extensions

✅ Congratulations, you've successfully set up the development environment. However, we highly recommend you take the time to learn how to set up a connection with the department machines. If there are any bugs in the container later on, this will allow you to continue developing without interruption.

For these methods to work, you must set up remote access with the Brown CS system.

Follow this guide to develop remotely on department machines via a graphical interface. The advantages of this method is that you can edit your code using VSCode, run it, and debug it all on a department machine. We find that this method is the most straightforward to work with. However, the GUI can be slow at times, especially if you're working off-campus over VPN.

Another option is to use VSCode to connect to the department machines through ssh, more instructions of which can be found at the bottom of this page. Note that if developing Weenix (or any project that needs X11 Forwarding), the specific cluster that you are connecting to does not support X Forwarding. So while you are able to have your changes reflected on the department file system and edit it from your local machine, to run your code, you must open a separate terminal window and SSH with X forwarding enabled into the department machines and run Weenix there. (./weenix -n)

If the guide above does not work, we would recommend editing your configuration file. Open the command palette by navigating to View > Command Palette and type Remote-SSH: Open Configuration File. Add the configuration below to the configuration file that exists in the ~/.ssh directory.

Host BrownCS
  HostName fastx-cluster.cs.brown.edu
  User <cs_login>
  ConnectTimeout 60

Follow this guide. You can develop locally, but you will have to ssh through a terminal window locally to run your code (and transfer your code using scp or Github to department machines). You will need to forward X windows but only when spawning the emulator for Weenix (QEMU). To do so, you can run ssh <cslogin>@ssh.cs.brown.edu -Y. The -Y flag is case sensitive.

Follow this guide. We highly recommend you also setup pageant. If you are running Weenix, you will need to use the MobaXterm instructions on the guide in order to forward X windows.

Follow this guide to mount the department filesystem, so that you can edit you files. You will however still need to ssh into the department machines to run your code.

Note: Add the following to your ssh config file to improve the speed of X11 Forwarding

Host ssh.cs.brown.edu
  HostName ssh.cs.brown.edu
  User <cslogin>
  ObscureKeystrokeTiming no
  ForwardX11Trusted yes

⚠️ We have not tested these methods in a while, so we cannot guarantee they still work. We strongly recommend using the container instead, or developing remotely on the department machines if you encounter issues with the former setup.

If you run Linux on your laptop, you may be able to install QEMU and other dependencies to build and run Weenix. Run the following command to do so:

sudo apt-get install gcc gdb qemu-system make python3 python3-pyelftools cscope xterm bash grub-pc-bin xorriso mtools

If a package that is being installed is not available, please make a post on EdStem!

If you use any other setup, we may not be able to provide support in resolving any issues that you might come across.

If you want to work on Weenix on your own machine and not have to create an ssh connection or depend on the department machines to run your code, we recommend using Vagrant. Using Vagrant and the Vagrantfile provided in the directory you cloned, you can spawn a VM running Linux with all dependencies necessary for running Weenix. Because VirtualBox does not support virtualization for ARM processors, those who are using M1s and M2s will not be able to use this setup, please see the section on setting up the container or working from department machines.

To get this working:

1. Install an X server and an SSH client.

• For Mac, this will be XQuartz.
• For Windows, this will be PuTTY and MobaXTerm.

2. Clone your assignment repository from GitHub Classroom onto your computer and cd into the directory.
3. Follow the instructions to setup Vagrant here—provided from cs33.

Please use the Vagrantfile in the assignment stencil or the one below, instead of the one provided in the guide.

VAGRANT_BOX_MEMORY = 4096
VAGRANT_BOX_CPUS = 4
Vagrant.configure("2") do |config|
  # Ubuntu 18
  config.vm.box = "ubuntu/bionic64"
  config.ssh.forward_x11 = true
  # Set the Vagrant box's RAM
  config.vm.provider :virtualbox do |vb|
    vb.memory = VAGRANT_BOX_MEMORY
    vb.cpus = VAGRANT_BOX_CPUS
  end
  # Add to the vagrant box's provisioning script: install dependencies for weenix
  config.vm.provision :shell, :inline => %Q{
    sudo apt-get update
    sudo apt-get install python2.7 python-minimal cscope nasm make build-essential grub2-common qemu xorriso genisoimage xterm gdb -y
  }
end

4. In your assignment repository on your home computer, run vagrant up from the terminal on macOS or Linux, or from git bash on Windows. Vagrant will traverse up until it finds the Vagrantfile at the root of the repository, create a VM with the correct specifications, and launch it in the background (headless). This will take some time. After these steps, your virtual machine will be setup.

5. To ssh into the vagrant VM to run your code use the following command vagrant ssh
   • (Weenix) When running Weenix, it is important that when you ssh, you enable X forwarding with the following command: vagrant ssh -- -Y. This command is case sensitive.
6. Once you're SSH'd in, use the command cd /vagrant to enter the shared directory. You can compile and run your code from here.

5. Watch the following YouTube video, which has an explanation for how to connect to your VM with PuTTY.
6. Edit the PuTTY session settings to enable X11 Forwarding: in the "PuTTY Configuration" window, Category > Connection > SSH > X11 > Enable X11 Forwarding.
7. Run the vagrant VM by double clicking on the saved PuTTY vagrant session that was just created.
8. Once you're SSH'd in, if you are not already in your shared Weenix directory. Use the command cd /vagrant to enter the shared directory. You can compile and run your code from here.

When you're done working, remember to run vagrant suspend or vagrant halt from the same place you ran vagrant up, otherwise your VM will continue running in the background.

The commands that you will use most frequently are the following:

• vagrant up will turn on your VM
• vagrant halt will shut down your VM (you will have to run vagrant up in order to use your VM)
• vagrant ssh allows you to connect to your VM
• vagrant status allows you to check the status of your VM
• To see more commands for vagrant, please see the following resource.

If you’re on a Mac and VirtualBox fails to install, follow these steps to change your security settings in Systems Preferences.

If running vagrant up times out, we recommend trying the following steps:

• Make sure that intel virtualization has been enabled (Windows specific)
• Run vagrant destroy, followed by vagrant up
• Make sure that the version of vagrant that you are using is up to date and turn on cable connect for the VM. You can do this by going into the VirtualBox application, selecting the VM, navigating to Settings > Network > Advanced, and turning cable connect on.

As always, if you run into any questions or problems, please don’t hesitate to reach out to us on EdStem.

• If your cursor gets stuck within QEMU, you can hold CTRL + ALT to regain full control of the curson and move it outside of QEMU.

• If you see the following error after running ./weenix -n:

QEMU error: Could not initialize SDL(No available video device) - exiting

Make sure that you have both:

1. installed an X server as described in step 3 of the "Vagrant" section.
2. if running Weenix using a VM, that you have ssh'ed with vagrant ssh -- -Y.

If none of the above suggestions help, please feel free to make a post on EdStem!

• If you see the following error after running ./weenix -n:

-bash: ./weenix: Permission denied

Run chmod +x weenix to fix it.

• If you see the following error after running ./weenix -n:

-bash: ./weenix: /bin/bash^M: bad interpreter: No such file or directory

Run dos2unix weenix to fix it.

• If you are using SSH and see the following issue when trying to gdb:

preinit.gdb:1: Error in sourced command file:
localhost:1234: Cannot assign requested address.
Breakpoint 1 at 0xffff80000011278b: file util/debug.c, line 216.
Breakpoint 2 at 0xffff8000001000e6: file entry/entry.c, line 10.
init.gdb:40: Error in sourced command file:
The program is not being run.

Make sure you are connected to Brown wifi. If you are unable to connect to Brown wifi, please use FastX instead.

• Make sure to close all QEMU windows before re-running Weenix as previous sessions can interfere.

• To avoid symlink issues that occur in VM for students who use Windows, it is recommended that files are pushed and pulled through GitHub rather than using WinSCP to transfer files from home computers to department machines.