AWS Setup Instructions - l-nic/chipyard GitHub Wiki

In order to experiment with the nanoPU artifact you will need access to an AWS account and you will need to configure that account to run Firesim simulations.

The instructions provided on this page are adapted (almost verbatim) from the Firesim documentation. We include it here for convenience, clarity, and also just in case the Firesim documentation changes in the future.

First-time AWS User Setup

If you’ve never used AWS before and don’t have an account, follow the instructions below to get started.

Creating an AWS Account

First, you’ll need an AWS account. Create one by going to aws.amazon.com and clicking “Sign Up.” You will have to give it a credit card number.

Requesting Limit Increases

In our experience, new AWS accounts do not have access to EC2 F1 instances by default. In order to get access, you should file a limit increase request. You can learn more about EC2 instance limits here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-on-demand-instances.html#ec2-on-demand-instances-limits

To request a limit increase, follow these steps: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html

You’ll probably want to start out with the following request, depending on your existing limits:

Limit Type:                EC2 Instances
Region:                    US West (Oregon)
Primary Instance Type:     All F instances
Limit:                     Instance Limit
New limit value:           64

For the region, you should select one of: us-east-1 (N. Virginia), us-west-2 (Oregon), or eu-west-1 (Ireland), since F1 instances are only available in those regions.

For the “Use Case Description”, you should describe your project and write something about hardware simulation and mention that information about the tool you’re using can be found at: https://fires.im

This process has a human in the loop, so you should submit it ASAP. You'll need to wait for the request to be approved before you can run the nanoPU Firesim Simulations.

Do not be surprised if the request is only partially granted. You only actually need 8 vCPUs to run the nanoPU Firesim simulations. 8 vCPUs is enough to launch one f1.2xlarge instance, which has one VU9P FPGA and can be used to simulate one nanoPU node.

Configuring Required Infrastructure in Your AWS Account

Once we have an AWS Account setup, we need to perform some advance setup of resources on AWS. You will need to follow these steps even if you already had an AWS account as these are FireSim-specific.

Select a Region

Head to the EC2 Management Console. In the top right corner, ensure that the correct region is selected. You should select one of: us-east-1 (N. Virginia), us-west-2 (Oregon), or eu-west-1 (Ireland), since F1 instances are only available in those regions. Make sure to select the same region that you used in your request to access F1 instances!

Once you select a region, it’s useful to bookmark the link to the EC2 console, so that you’re always sent to the console for the correct region.

Key Setup

In order to enable automation, you will need to create a key named firesim, which we will use to launch all instances (Manager Instance, Build Farm, Run Farm).

To do so, click “Key Pairs” under “Network & Security” in the left-sidebar. Follow the prompts, name the key firesim, and save the private key locally as firesim.pem. You can use this key to access all instances from your local machine. We will copy this file to our manager instance later, so that the manager can also use it.

Check your EC2 Instance Limits

AWS limits access to particular instance types for new/infrequently used accounts to protect their infrastructure. You should make sure that your account has access to f1.2xlarge and c5.4xlarge instances by looking at the “Limits” page in the EC2 panel, which you can access here. The values listed on this page represent the maximum number of any of these instances that you can run at once, which will limit the size of simulations (# of nodes) that you can run. If you need to increase your limits, follow the instructions described earlier on this page. To run the nanoPU Firesim Simulations you will need to be able to run one f1.2xlarge instance and one c5.4xlarge instances. If you would like to build a custom FPGA image for Firesim simulations then you will need to be able to run two c5.4xlarge instances.

Start a t2.nano instance to run the remaining configuration commands

To avoid having to deal with the messy process of installing packages on your local machine, we will spin up a very cheap t2.nano instance to run a series of one-time aws configuration commands to setup our AWS account for FireSim. At the end of these instructions, we’ll terminate the t2.nano instance. If you happen to already have boto3 and the AWS CLI installed on your local machine, you can do this locally.

Launch a t2.nano by following these instructions:

  1. Go to the EC2 Management Console and click “Launch Instance”.
  2. On the AMI selection page, select “Amazon Linux AMI...”, which should be the top option.
  3. On the Choose an Instance Type page, select t2.nano.
  4. Click “Review and Launch” (we don’t need to change any other settings)
  5. On the review page, click “Launch”
  6. Select the firesim key pair we created previously, then click Launch Instances.
  7. Click on the instance name and note its public IP address.

Run scripts from the t2.nano

SSH into the t2.nano like so:

ssh -i firesim.pem ec2-user@INSTANCE_PUBLIC_IP

Which should present you with something like:

Last login: Mon Feb 12 21:11:27 2018 from 136.152.143.34

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2017.09-release-notes/
4 package(s) needed for security, out of 5 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-30-2-66 ~]$

On this machine, run the following:

aws configure
[follow prompts]

See https://docs.aws.amazon.com/cli/latest/userguide/tutorial-ec2-ubuntu.html#configure-cli-launch-ec2 for more about aws configure. Within the prompt, you should specify the same region that you chose above (one of us-east-1, us-west-2, eu-west-1) and set the default output format to json. You will need to generate an AWS access key in the “Security Credentials” menu of your AWS settings (as instructed in https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys ).

Again on the t2.nano instance, do the following:

sudo yum -y install python-pip
sudo pip install boto3
sudo pip install --upgrade awscli
wget https://raw.githubusercontent.com/l-nic/firesim/lnic-dev/scripts/aws-setup.py
python aws-setup.py

This will create a VPC named firesim and a security group named firesim in your account.

Terminate the t2.nano

At this point, we are finished with the general account configuration. You should terminate the t2.nano instance you created, since we do not need it anymore (and it shouldn’t contain any important data).

Subscribe to the AWS FPGA Developer AMI

Go to the AWS Marketplace page for the FPGA Developer AMI. Click the button to subscribe to the FPGA Dev AMI (it should be free) and follow the prompts to accept the EULA (but do not launch any instances).

Setting up your Manager Instance

We have created a custom Amazon Machine Image (AMI) that has the necessary tools and repositories needed to run all the nanoPU simulations. You should use this AMI for your Firesim manager instance. Unfortunately, Amazon will not allow us to make the image public on the Community market place because it uses a number of licensed tools. Therefore, please send an email to sibanez [at] stanford.edu and indicate your numeric AWS account ID. You can use the following email template:

Subject: nanoPU Artifact AMI

Hi,

I would like to gain access to the nanoPU Artifact AMI.
My AWS account ID is: XXXXXXXXXXXX

Thank you.

We will share the AMI with you as soon as possible.

Launching a “Manager Instance”

Now, we need to launch a “Manager Instance” that acts as a “head” node that we will ssh or mosh into to work from. We will use a c5.4xlarge, running the our custom nanoPU artifact AMI.

Head to the EC2 Management Console. In the top right corner, ensure that the correct region is selected.

To launch a manager instance, follow these steps:

  1. From the main page of the EC2 Management Console, click Launch Instance. We use an on-demand instance here, so that your data is preserved when you stop/start the instance, and your data is not lost when pricing spikes on the spot market.
  2. When prompted to select an AMI, search in the "My AMIs" tab for "nanoPU Artifact v1.0".
  3. When prompted to choose an instance type, select the instance type of your choosing. A good choice is a c5.4xlarge.
  4. On the “Configure Instance Details” page:
  • First make sure that the firesim VPC is selected in the drop-down box next to “Network”. Any subnet within the firesim VPC is fine.
  • Additionally, check the box for “Protect against accidental termination.” This adds a layer of protection to prevent your manager instance from being terminated by accident. You will need to disable this setting before being able to terminate the instance using usual methods.
  • On the next page (“Add Storage”), make sure the size of the root EBS volume is ~256GB. Vivado reports/outputs and large simulation waveforms consume a lot of space.
  • You can skip the “Add Tags” page, unless you want tags.
  • On the “Configure Security Group” page, select the firesim security group that was automatically created for you earlier.
  • On the review page, click the button to launch your instance.

Make sure you select the firesim key pair that we setup earlier.

Access your instance

We HIGHLY recommend using either ssh with a screen/tmux session running on your manager instance or mosh to ensure that long-running jobs are not killed by a bad network connection to your manager instance.

ssh into your manage instance like so:

ssh -i firesim.pem -L 8888:localhost:8888 centos@YOUR_INSTANCE_IP

Note that we need to use the firesim.pem key that we created earlier and we are forwarding port 8888 over the ssh connection so that we can access the Jupyter notebook server that we will run on the manager instance to view the simulation results.

Now that our manager instance is started, copy the private key that you downloaded from AWS earlier (firesim.pem) to ~/firesim.pem on your manager instance. This step is required to give the manager access to the instances it launches for you.

Set up the evironment

Run the following:

$ cd chipyard/sims/firesim
$ source sourceme-f1-manager.sh

This will have initialized the AWS shell, added the RISC-V tools to your path, and started an ssh-agent that supplies ~/firesim.pem automatically when you use ssh to access other nodes. Sourcing this the first time will take some time – however each time after that should be instantaneous. Also, if your firesim.pem key requires a passphrase, you will be asked for it here and ssh-agent should cache it.

Every time you login to your manager instance you should cd into your firesim directory and source this file again.

Completing Setup Using the Manager

The FireSim manager contains a command that will interactively guide you through the rest of the FireSim setup process. To run it, do the following:

firesim managerinit

This will first prompt you to setup AWS credentials on the instance, which allows the manager to automatically manage build/simulation nodes. See https://docs.aws.amazon.com/cli/latest/userguide/tutorial-ec2-ubuntu.html#configure-cli-launch-ec2 for more about these credentials. When prompted, you should specify the same region that you chose above and set the default output format to json.

Next, it will create initial configuration files, which we will overwrite shortly. Finally, it will prompt you for an email address, which is used to send email notifications upon FPGA build completion and optionally for workload completion. You can leave this blank if you do not wish to receive any notifications, but this is not recommended.

Run the following commands to overwrite the default Firesim config files with ones that we will use for our nanoPU simulations:

cd ~/chipyard/sims/firesim/deploy/
./setup-nanoPU-config.sh

Now you’re ready to launch nanoPU simulations!